Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 08:16:34.796396 6164 test_util.cc:268] Using random seed: 950887237
I20240502 08:16:34.807241 6164 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:16:34.807315 6164 ts_itest-base.cc:113] --------------
I20240502 08:16:34.807356 6164 ts_itest-base.cc:114] 5 tablet servers
I20240502 08:16:34.807396 6164 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:16:34.807440 6164 ts_itest-base.cc:116] --------------
2024-05-02T08:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:16:34Z Disabled control of system clock
I20240502 08:16:34.828043 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.5.62:40289
--webserver_interface=127.6.5.62
--webserver_port=0
--builtin_ntp_servers=127.6.5.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.5.62:40289 with env {}
W20240502 08:16:35.128705 6176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:35.128980 6176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:35.129096 6176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:35.136960 6176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:35.137075 6176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:35.137151 6176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:35.137233 6176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:35.146360 6176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.5.62:40289
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.5.62:40289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.5.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:35.147534 6176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:35.149209 6176 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:35.159642 6182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.160091 6184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.160459 6181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.160996 6176 server_base.cc:1034] running on GCE node
I20240502 08:16:35.161666 6176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:35.162935 6176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:35.164165 6176 hybrid_clock.cc:648] HybridClock initialized: now 1714637795164081 us; error 60 us; skew 500 ppm
I20240502 08:16:35.164474 6176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:35.171799 6176 webserver.cc:460] Webserver started at http://127.6.5.62:42613/ using document root <none> and password file <none>
I20240502 08:16:35.172468 6176 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:35.172588 6176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:35.172873 6176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:35.175472 6176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c4fa09e5a45740698fe844f76476f924"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-h7h6"
I20240502 08:16:35.176160 6176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c4fa09e5a45740698fe844f76476f924"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-h7h6"
I20240502 08:16:35.180845 6176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:16:35.183907 6190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.185099 6176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:35.185253 6176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c4fa09e5a45740698fe844f76476f924"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-h7h6"
I20240502 08:16:35.185412 6176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:35.216972 6176 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:35.217731 6176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:35.218008 6176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:35.234556 6176 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.62:40289
I20240502 08:16:35.234598 6241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.62:40289 every 8 connection(s)
I20240502 08:16:35.235800 6176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:16:35.238723 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 6176
I20240502 08:16:35.238953 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:16:35.239362 6242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:35.248945 6242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924: Bootstrap starting.
I20240502 08:16:35.251627 6242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:35.252605 6242 log.cc:824] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:35.255029 6242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924: No bootstrap required, opened a new log
I20240502 08:16:35.261046 6242 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fa09e5a45740698fe844f76476f924" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 40289 } }
I20240502 08:16:35.261356 6242 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:35.261461 6242 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fa09e5a45740698fe844f76476f924, State: Initialized, Role: FOLLOWER
I20240502 08:16:35.262113 6242 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fa09e5a45740698fe844f76476f924" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 40289 } }
I20240502 08:16:35.262316 6242 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:35.262444 6242 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:35.262627 6242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:35.265007 6242 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fa09e5a45740698fe844f76476f924" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 40289 } }
I20240502 08:16:35.265462 6242 leader_election.cc:302] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4fa09e5a45740698fe844f76476f924; no voters:
I20240502 08:16:35.265960 6242 leader_election.cc:288] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:35.266093 6247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:35.266671 6247 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [term 1 LEADER]: Becoming Leader. State: Replica: c4fa09e5a45740698fe844f76476f924, State: Running, Role: LEADER
I20240502 08:16:35.267166 6247 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fa09e5a45740698fe844f76476f924" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 40289 } }
I20240502 08:16:35.267778 6242 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:35.269663 6248 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4fa09e5a45740698fe844f76476f924" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fa09e5a45740698fe844f76476f924" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 40289 } } }
I20240502 08:16:35.269954 6248 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:35.272641 6249 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4fa09e5a45740698fe844f76476f924. Latest consensus state: current_term: 1 leader_uuid: "c4fa09e5a45740698fe844f76476f924" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fa09e5a45740698fe844f76476f924" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 40289 } } }
I20240502 08:16:35.273003 6249 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:35.273586 6255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:35.278367 6255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:35.286031 6255 catalog_manager.cc:1347] Generated new cluster ID: 34ae867326074e879934cca2fdbb1a3f
I20240502 08:16:35.286128 6255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:35.310645 6255 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:35.311852 6255 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:35.319365 6255 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c4fa09e5a45740698fe844f76476f924: Generated new TSK 0
I20240502 08:16:35.320372 6255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:35.337756 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.1:0
--local_ip_for_outbound_sockets=127.6.5.1
--webserver_interface=127.6.5.1
--webserver_port=0
--tserver_master_addrs=127.6.5.62:40289
--builtin_ntp_servers=127.6.5.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240502 08:16:35.642297 6266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:35.642580 6266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:35.642716 6266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:35.649672 6266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:35.649848 6266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.1
I20240502 08:16:35.658083 6266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.5.1
--webserver_port=0
--tserver_master_addrs=127.6.5.62:40289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.1
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:35.659201 6266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:35.660744 6266 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:35.672094 6271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.672528 6272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:35.672935 6274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:35.673418 6266 server_base.cc:1034] running on GCE node
I20240502 08:16:35.673918 6266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:35.674754 6266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:35.675948 6266 hybrid_clock.cc:648] HybridClock initialized: now 1714637795675883 us; error 41 us; skew 500 ppm
I20240502 08:16:35.676209 6266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:35.678150 6266 webserver.cc:460] Webserver started at http://127.6.5.1:32903/ using document root <none> and password file <none>
I20240502 08:16:35.678813 6266 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:35.678928 6266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:35.679195 6266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:35.681459 6266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2b2891c95b61485ba6c078bdf50d2621"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-h7h6"
I20240502 08:16:35.682114 6266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2b2891c95b61485ba6c078bdf50d2621"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-h7h6"
I20240502 08:16:35.686597 6266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20240502 08:16:35.689574 6280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.690779 6266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:35.690934 6266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2b2891c95b61485ba6c078bdf50d2621"
format_stamp: "Formatted at 2024-05-02 08:16:35 on dist-test-slave-h7h6"
I20240502 08:16:35.691114 6266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:35.716349 6266 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:35.717068 6266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:35.717317 6266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:35.718138 6266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:35.719516 6266 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:35.719585 6266 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.719694 6266 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:35.719743 6266 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:35.747649 6266 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.1:37931
I20240502 08:16:35.747720 6392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.1:37931 every 8 connection(s)
I20240502 08:16:35.748792 6266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:16:35.752480 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 6266
I20240502 08:16:35.752729 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:16:35.755726 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.2:0
--local_ip_for_outbound_sockets=127.6.5.2
--webserver_interface=127.6.5.2
--webserver_port=0
--tserver_master_addrs=127.6.5.62:40289
--builtin_ntp_servers=127.6.5.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 08:16:35.760581 6393 heartbeater.cc:347] Connected to a master server at 127.6.5.62:40289
I20240502 08:16:35.760830 6393 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:35.761543 6393 heartbeater.cc:511] Master 127.6.5.62:40289 requested a full tablet report, sending...
I20240502 08:16:35.763396 6207 ts_manager.cc:194] Registered new tserver with Master: 2b2891c95b61485ba6c078bdf50d2621 (127.6.5.1:37931)
I20240502 08:16:35.764931 6207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.1:43415
W20240502 08:16:36.056126 6396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:36.056391 6396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:36.056506 6396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:36.063530 6396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:36.063709 6396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.2
I20240502 08:16:36.072240 6396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.5.2
--webserver_port=0
--tserver_master_addrs=127.6.5.62:40289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.2
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:36.073365 6396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:36.074995 6396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:36.086251 6401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.086733 6402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.087100 6404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:36.087613 6396 server_base.cc:1034] running on GCE node
I20240502 08:16:36.088146 6396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:36.088984 6396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:36.090194 6396 hybrid_clock.cc:648] HybridClock initialized: now 1714637796090121 us; error 37 us; skew 500 ppm
I20240502 08:16:36.090495 6396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:36.092505 6396 webserver.cc:460] Webserver started at http://127.6.5.2:38861/ using document root <none> and password file <none>
I20240502 08:16:36.093108 6396 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:36.093220 6396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:36.093472 6396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:36.095938 6396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ae436285fe034bc9b1add77863fd1d57"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-h7h6"
I20240502 08:16:36.096589 6396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ae436285fe034bc9b1add77863fd1d57"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-h7h6"
I20240502 08:16:36.101167 6396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:16:36.104223 6410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.105377 6396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:36.105502 6396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ae436285fe034bc9b1add77863fd1d57"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-h7h6"
I20240502 08:16:36.105648 6396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:36.120748 6396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:36.121384 6396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:36.121613 6396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:36.122381 6396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:36.123813 6396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:36.123885 6396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.123968 6396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:36.124011 6396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.151504 6396 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.2:42769
I20240502 08:16:36.151553 6522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.2:42769 every 8 connection(s)
I20240502 08:16:36.152638 6396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:16:36.159018 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 6396
I20240502 08:16:36.159202 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:16:36.162410 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.3:0
--local_ip_for_outbound_sockets=127.6.5.3
--webserver_interface=127.6.5.3
--webserver_port=0
--tserver_master_addrs=127.6.5.62:40289
--builtin_ntp_servers=127.6.5.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 08:16:36.164592 6523 heartbeater.cc:347] Connected to a master server at 127.6.5.62:40289
I20240502 08:16:36.164870 6523 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:36.165715 6523 heartbeater.cc:511] Master 127.6.5.62:40289 requested a full tablet report, sending...
I20240502 08:16:36.167035 6207 ts_manager.cc:194] Registered new tserver with Master: ae436285fe034bc9b1add77863fd1d57 (127.6.5.2:42769)
I20240502 08:16:36.167943 6207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.2:38969
W20240502 08:16:36.469475 6526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:36.469722 6526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:36.469820 6526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:36.477361 6526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:36.477522 6526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.3
I20240502 08:16:36.486235 6526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.5.3
--webserver_port=0
--tserver_master_addrs=127.6.5.62:40289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.3
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:36.487370 6526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:36.489017 6526 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:36.500769 6531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.501282 6532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.501770 6534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:36.502279 6526 server_base.cc:1034] running on GCE node
I20240502 08:16:36.502848 6526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:36.503634 6526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:36.504827 6526 hybrid_clock.cc:648] HybridClock initialized: now 1714637796504774 us; error 29 us; skew 500 ppm
I20240502 08:16:36.505077 6526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:36.507055 6526 webserver.cc:460] Webserver started at http://127.6.5.3:41127/ using document root <none> and password file <none>
I20240502 08:16:36.507649 6526 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:36.507759 6526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:36.508008 6526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:36.510368 6526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a13035871a8b43339580871e0da23720"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-h7h6"
I20240502 08:16:36.511025 6526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a13035871a8b43339580871e0da23720"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-h7h6"
I20240502 08:16:36.515481 6526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:16:36.518378 6540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.519541 6526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:16:36.519673 6526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a13035871a8b43339580871e0da23720"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-h7h6"
I20240502 08:16:36.519824 6526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:36.532269 6526 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:36.532898 6526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:36.533134 6526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:36.533912 6526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:36.535214 6526 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:36.535282 6526 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.535393 6526 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:36.535442 6526 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.563266 6526 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.3:32919
I20240502 08:16:36.563320 6652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.3:32919 every 8 connection(s)
I20240502 08:16:36.564342 6526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:16:36.566975 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 6526
I20240502 08:16:36.567152 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:16:36.571013 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.4:0
--local_ip_for_outbound_sockets=127.6.5.4
--webserver_interface=127.6.5.4
--webserver_port=0
--tserver_master_addrs=127.6.5.62:40289
--builtin_ntp_servers=127.6.5.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 08:16:36.577368 6653 heartbeater.cc:347] Connected to a master server at 127.6.5.62:40289
I20240502 08:16:36.577643 6653 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:36.578389 6653 heartbeater.cc:511] Master 127.6.5.62:40289 requested a full tablet report, sending...
I20240502 08:16:36.579825 6207 ts_manager.cc:194] Registered new tserver with Master: a13035871a8b43339580871e0da23720 (127.6.5.3:32919)
I20240502 08:16:36.580854 6207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.3:49545
I20240502 08:16:36.767428 6393 heartbeater.cc:503] Master 127.6.5.62:40289 was elected leader, sending a full tablet report...
W20240502 08:16:36.872002 6656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:36.872234 6656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:36.872359 6656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:36.880324 6656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:36.880483 6656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.4
I20240502 08:16:36.888751 6656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.5.4
--webserver_port=0
--tserver_master_addrs=127.6.5.62:40289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.4
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:36.889797 6656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:36.891363 6656 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:36.902611 6661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.903062 6662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:36.903538 6664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:36.904037 6656 server_base.cc:1034] running on GCE node
I20240502 08:16:36.904599 6656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:36.905392 6656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:36.906549 6656 hybrid_clock.cc:648] HybridClock initialized: now 1714637796906496 us; error 27 us; skew 500 ppm
I20240502 08:16:36.906808 6656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:36.908787 6656 webserver.cc:460] Webserver started at http://127.6.5.4:44097/ using document root <none> and password file <none>
I20240502 08:16:36.909488 6656 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:36.909610 6656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:36.909879 6656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:36.912310 6656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6b2832dce0714b4f8bd62acdd56a6555"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-h7h6"
I20240502 08:16:36.913029 6656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6b2832dce0714b4f8bd62acdd56a6555"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-h7h6"
I20240502 08:16:36.917637 6656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:16:36.920691 6670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.921851 6656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:36.921993 6656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6b2832dce0714b4f8bd62acdd56a6555"
format_stamp: "Formatted at 2024-05-02 08:16:36 on dist-test-slave-h7h6"
I20240502 08:16:36.922148 6656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:36.937709 6656 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:36.938370 6656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:36.938647 6656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:36.939457 6656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:36.940788 6656 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:36.940865 6656 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.940977 6656 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:36.941030 6656 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:36.971319 6656 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.4:35443
I20240502 08:16:36.971380 6782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.4:35443 every 8 connection(s)
I20240502 08:16:36.972438 6656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:16:36.976946 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 6656
I20240502 08:16:36.977123 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:16:36.979816 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.5:0
--local_ip_for_outbound_sockets=127.6.5.5
--webserver_interface=127.6.5.5
--webserver_port=0
--tserver_master_addrs=127.6.5.62:40289
--builtin_ntp_servers=127.6.5.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 08:16:36.984087 6783 heartbeater.cc:347] Connected to a master server at 127.6.5.62:40289
I20240502 08:16:36.984447 6783 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:36.985162 6783 heartbeater.cc:511] Master 127.6.5.62:40289 requested a full tablet report, sending...
I20240502 08:16:36.986398 6207 ts_manager.cc:194] Registered new tserver with Master: 6b2832dce0714b4f8bd62acdd56a6555 (127.6.5.4:35443)
I20240502 08:16:36.987191 6207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.4:39579
I20240502 08:16:37.170305 6523 heartbeater.cc:503] Master 127.6.5.62:40289 was elected leader, sending a full tablet report...
W20240502 08:16:37.282649 6786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:37.282903 6786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:37.283030 6786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:37.290246 6786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:37.290418 6786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.5
I20240502 08:16:37.298895 6786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.5.5
--webserver_port=0
--tserver_master_addrs=127.6.5.62:40289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.5
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:37.300006 6786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:37.301573 6786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:37.313120 6791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:37.313642 6792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:37.314105 6794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:37.314518 6786 server_base.cc:1034] running on GCE node
I20240502 08:16:37.315083 6786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:37.315871 6786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:37.317082 6786 hybrid_clock.cc:648] HybridClock initialized: now 1714637797317012 us; error 40 us; skew 500 ppm
I20240502 08:16:37.317353 6786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:37.319370 6786 webserver.cc:460] Webserver started at http://127.6.5.5:39103/ using document root <none> and password file <none>
I20240502 08:16:37.320005 6786 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:37.320124 6786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:37.320398 6786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:37.322753 6786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "5d45cd9deaa2420e8d6d8a9add334d63"
format_stamp: "Formatted at 2024-05-02 08:16:37 on dist-test-slave-h7h6"
I20240502 08:16:37.323446 6786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "5d45cd9deaa2420e8d6d8a9add334d63"
format_stamp: "Formatted at 2024-05-02 08:16:37 on dist-test-slave-h7h6"
I20240502 08:16:37.328114 6786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:16:37.331068 6800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:37.332212 6786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:37.332353 6786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5d45cd9deaa2420e8d6d8a9add334d63"
format_stamp: "Formatted at 2024-05-02 08:16:37 on dist-test-slave-h7h6"
I20240502 08:16:37.332507 6786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:37.352946 6786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:37.353613 6786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:37.353859 6786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:37.354671 6786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:37.356000 6786 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:37.356072 6786 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:37.356182 6786 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:37.356232 6786 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:37.383539 6786 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.5:41605
I20240502 08:16:37.383577 6912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.5:41605 every 8 connection(s)
I20240502 08:16:37.384651 6786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:16:37.393348 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 6786
I20240502 08:16:37.393540 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:16:37.396020 6913 heartbeater.cc:347] Connected to a master server at 127.6.5.62:40289
I20240502 08:16:37.396292 6913 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:37.397166 6913 heartbeater.cc:511] Master 127.6.5.62:40289 requested a full tablet report, sending...
I20240502 08:16:37.398352 6207 ts_manager.cc:194] Registered new tserver with Master: 5d45cd9deaa2420e8d6d8a9add334d63 (127.6.5.5:41605)
I20240502 08:16:37.399101 6207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.5:38075
I20240502 08:16:37.409449 6164 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:16:37.428411 6207 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50930:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 08:16:37.458896 6328 tablet_service.cc:1467] Processing CreateTablet for tablet 41a5402c453b4a70badab626ddf912f9 (DEFAULT_TABLE table=TestTable [id=3b17d5ba5e374990b03d79295d79c093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.460342 6328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a5402c453b4a70badab626ddf912f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.461676 6848 tablet_service.cc:1467] Processing CreateTablet for tablet 41a5402c453b4a70badab626ddf912f9 (DEFAULT_TABLE table=TestTable [id=3b17d5ba5e374990b03d79295d79c093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.463199 6848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a5402c453b4a70badab626ddf912f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.464749 6458 tablet_service.cc:1467] Processing CreateTablet for tablet 41a5402c453b4a70badab626ddf912f9 (DEFAULT_TABLE table=TestTable [id=3b17d5ba5e374990b03d79295d79c093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.466212 6458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a5402c453b4a70badab626ddf912f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.470515 6931 tablet_bootstrap.cc:492] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621: Bootstrap starting.
I20240502 08:16:37.473479 6931 tablet_bootstrap.cc:654] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.474558 6931 log.cc:824] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:37.476603 6933 tablet_bootstrap.cc:492] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63: Bootstrap starting.
I20240502 08:16:37.476995 6931 tablet_bootstrap.cc:492] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621: No bootstrap required, opened a new log
I20240502 08:16:37.477243 6931 ts_tablet_manager.cc:1367] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 08:16:37.479259 6934 tablet_bootstrap.cc:492] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57: Bootstrap starting.
I20240502 08:16:37.480163 6933 tablet_bootstrap.cc:654] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.481382 6933 log.cc:824] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:37.481556 6934 tablet_bootstrap.cc:654] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.482599 6934 log.cc:824] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:37.484099 6933 tablet_bootstrap.cc:492] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63: No bootstrap required, opened a new log
I20240502 08:16:37.484364 6933 ts_tablet_manager.cc:1367] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 08:16:37.484856 6934 tablet_bootstrap.cc:492] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57: No bootstrap required, opened a new log
I20240502 08:16:37.485060 6934 ts_tablet_manager.cc:1367] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:16:37.485524 6931 raft_consensus.cc:348] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2891c95b61485ba6c078bdf50d2621" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 37931 } } peers { permanent_uuid: "ae436285fe034bc9b1add77863fd1d57" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 42769 } } peers { permanent_uuid: "5d45cd9deaa2420e8d6d8a9add334d63" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 41605 } }
I20240502 08:16:37.485976 6931 raft_consensus.cc:374] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.486086 6931 raft_consensus.cc:729] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b2891c95b61485ba6c078bdf50d2621, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.486781 6931 consensus_queue.cc:260] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2891c95b61485ba6c078bdf50d2621" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 37931 } } peers { permanent_uuid: "ae436285fe034bc9b1add77863fd1d57" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 42769 } } peers { permanent_uuid: "5d45cd9deaa2420e8d6d8a9add334d63" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 41605 } }
I20240502 08:16:37.491053 6934 raft_consensus.cc:348] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2891c95b61485ba6c078bdf50d2621" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 37931 } } peers { permanent_uuid: "ae436285fe034bc9b1add77863fd1d57" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 42769 } } peers { permanent_uuid: "5d45cd9deaa2420e8d6d8a9add334d63" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 41605 } }
I20240502 08:16:37.491350 6934 raft_consensus.cc:374] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.491441 6934 raft_consensus.cc:729] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae436285fe034bc9b1add77863fd1d57, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.491981 6934 consensus_queue.cc:260] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2891c95b61485ba6c078bdf50d2621" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 37931 } } peers { permanent_uuid: "ae436285fe034bc9b1add77863fd1d57" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 42769 } } peers { permanent_uuid: "5d45cd9deaa2420e8d6d8a9add334d63" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 41605 } }
I20240502 08:16:37.493279 6934 ts_tablet_manager.cc:1398] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240502 08:16:37.494874 6933 raft_consensus.cc:348] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2891c95b61485ba6c078bdf50d2621" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 37931 } } peers { permanent_uuid: "ae436285fe034bc9b1add77863fd1d57" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 42769 } } peers { permanent_uuid: "5d45cd9deaa2420e8d6d8a9add334d63" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 41605 } }
I20240502 08:16:37.495255 6933 raft_consensus.cc:374] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.495344 6933 raft_consensus.cc:729] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d45cd9deaa2420e8d6d8a9add334d63, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.495826 6931 ts_tablet_manager.cc:1398] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20240502 08:16:37.496047 6933 consensus_queue.cc:260] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2891c95b61485ba6c078bdf50d2621" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 37931 } } peers { permanent_uuid: "ae436285fe034bc9b1add77863fd1d57" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 42769 } } peers { permanent_uuid: "5d45cd9deaa2420e8d6d8a9add334d63" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 41605 } }
I20240502 08:16:37.497079 6913 heartbeater.cc:503] Master 127.6.5.62:40289 was elected leader, sending a full tablet report...
I20240502 08:16:37.497876 6933 ts_tablet_manager.cc:1398] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20240502 08:16:37.501132 6394 tablet.cc:2369] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:37.583160 6653 heartbeater.cc:503] Master 127.6.5.62:40289 was elected leader, sending a full tablet report...
I20240502 08:16:37.598906 6937 raft_consensus.cc:482] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.599220 6937 raft_consensus.cc:504] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2891c95b61485ba6c078bdf50d2621" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 37931 } } peers { permanent_uuid: "ae436285fe034bc9b1add77863fd1d57" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 42769 } } peers { permanent_uuid: "5d45cd9deaa2420e8d6d8a9add334d63" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 41605 } }
I20240502 08:16:37.601135 6937 leader_election.cc:288] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae436285fe034bc9b1add77863fd1d57 (127.6.5.2:42769), 5d45cd9deaa2420e8d6d8a9add334d63 (127.6.5.5:41605)
I20240502 08:16:37.608078 6478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a5402c453b4a70badab626ddf912f9" candidate_uuid: "2b2891c95b61485ba6c078bdf50d2621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae436285fe034bc9b1add77863fd1d57" is_pre_election: true
I20240502 08:16:37.608578 6478 raft_consensus.cc:2454] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b2891c95b61485ba6c078bdf50d2621 in term 0.
I20240502 08:16:37.608518 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a5402c453b4a70badab626ddf912f9" candidate_uuid: "2b2891c95b61485ba6c078bdf50d2621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d45cd9deaa2420e8d6d8a9add334d63" is_pre_election: true
I20240502 08:16:37.608986 6868 raft_consensus.cc:2454] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b2891c95b61485ba6c078bdf50d2621 in term 0.
I20240502 08:16:37.609143 6283 leader_election.cc:302] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b2891c95b61485ba6c078bdf50d2621, ae436285fe034bc9b1add77863fd1d57; no voters:
I20240502 08:16:37.609540 6937 raft_consensus.cc:2781] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:37.609705 6937 raft_consensus.cc:482] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:37.609831 6937 raft_consensus.cc:3037] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.611922 6937 raft_consensus.cc:504] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2891c95b61485ba6c078bdf50d2621" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 37931 } } peers { permanent_uuid: "ae436285fe034bc9b1add77863fd1d57" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 42769 } } peers { permanent_uuid: "5d45cd9deaa2420e8d6d8a9add334d63" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 41605 } }
I20240502 08:16:37.612449 6937 leader_election.cc:288] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [CANDIDATE]: Term 1 election: Requested vote from peers ae436285fe034bc9b1add77863fd1d57 (127.6.5.2:42769), 5d45cd9deaa2420e8d6d8a9add334d63 (127.6.5.5:41605)
I20240502 08:16:37.612793 6478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a5402c453b4a70badab626ddf912f9" candidate_uuid: "2b2891c95b61485ba6c078bdf50d2621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae436285fe034bc9b1add77863fd1d57"
I20240502 08:16:37.612982 6478 raft_consensus.cc:3037] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.612942 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a5402c453b4a70badab626ddf912f9" candidate_uuid: "2b2891c95b61485ba6c078bdf50d2621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d45cd9deaa2420e8d6d8a9add334d63"
I20240502 08:16:37.613140 6868 raft_consensus.cc:3037] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.615989 6478 raft_consensus.cc:2454] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b2891c95b61485ba6c078bdf50d2621 in term 1.
I20240502 08:16:37.615994 6868 raft_consensus.cc:2454] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b2891c95b61485ba6c078bdf50d2621 in term 1.
I20240502 08:16:37.616331 6283 leader_election.cc:302] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b2891c95b61485ba6c078bdf50d2621, ae436285fe034bc9b1add77863fd1d57; no voters:
I20240502 08:16:37.616648 6937 raft_consensus.cc:2781] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.617177 6937 raft_consensus.cc:686] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [term 1 LEADER]: Becoming Leader. State: Replica: 2b2891c95b61485ba6c078bdf50d2621, State: Running, Role: LEADER
I20240502 08:16:37.617676 6937 consensus_queue.cc:237] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2891c95b61485ba6c078bdf50d2621" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 37931 } } peers { permanent_uuid: "ae436285fe034bc9b1add77863fd1d57" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 42769 } } peers { permanent_uuid: "5d45cd9deaa2420e8d6d8a9add334d63" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 41605 } }
I20240502 08:16:37.623114 6205 catalog_manager.cc:5484] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b2891c95b61485ba6c078bdf50d2621 (127.6.5.1). New cstate: current_term: 1 leader_uuid: "2b2891c95b61485ba6c078bdf50d2621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2891c95b61485ba6c078bdf50d2621" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 37931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae436285fe034bc9b1add77863fd1d57" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 42769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d45cd9deaa2420e8d6d8a9add334d63" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 41605 } health_report { overall_health: UNKNOWN } } }
W20240502 08:16:37.636159 6914 tablet.cc:2369] T 41a5402c453b4a70badab626ddf912f9 P 5d45cd9deaa2420e8d6d8a9add334d63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:37.644595 6164 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:16:37.647503 6164 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2b2891c95b61485ba6c078bdf50d2621 to finish bootstrapping
W20240502 08:16:37.655006 6524 tablet.cc:2369] T 41a5402c453b4a70badab626ddf912f9 P ae436285fe034bc9b1add77863fd1d57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:37.656678 6164 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ae436285fe034bc9b1add77863fd1d57 to finish bootstrapping
I20240502 08:16:37.663486 6164 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a13035871a8b43339580871e0da23720 to finish bootstrapping
I20240502 08:16:37.671756 6164 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6b2832dce0714b4f8bd62acdd56a6555 to finish bootstrapping
I20240502 08:16:37.679284 6164 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5d45cd9deaa2420e8d6d8a9add334d63 to finish bootstrapping
I20240502 08:16:37.989409 6783 heartbeater.cc:503] Master 127.6.5.62:40289 was elected leader, sending a full tablet report...
I20240502 08:16:38.107424 6328 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:38.108796 6937 consensus_queue.cc:1035] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45cd9deaa2420e8d6d8a9add334d63" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 41605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:38.110637 6848 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:38.113031 6718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:38.120690 6588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:38.121592 6458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:38.146265 6937 consensus_queue.cc:1035] T 41a5402c453b4a70badab626ddf912f9 P 2b2891c95b61485ba6c078bdf50d2621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae436285fe034bc9b1add77863fd1d57" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:38.320286 6848 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:38.322156 6588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:38.328226 6718 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:38.330749 6328 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:38.340101 6458 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:38.475224 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 6266
I20240502 08:16:38.476581 6388 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:38.563091 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 6266
I20240502 08:16:38.584753 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 6396
I20240502 08:16:38.585690 6518 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:38.671128 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 6396
I20240502 08:16:38.685820 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 6526
I20240502 08:16:38.686878 6647 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:38.768211 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 6526
I20240502 08:16:38.781085 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 6656
I20240502 08:16:38.782058 6778 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:38.862298 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 6656
I20240502 08:16:38.875214 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 6786
I20240502 08:16:38.876257 6907 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:38.961915 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 6786
I20240502 08:16:38.975219 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 6176
I20240502 08:16:38.976316 6237 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:39.079933 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 6176
2024-05-02T08:16:39Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (4322 ms)
[----------] 1 test from AdminCliTest (4322 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240502 08:16:39.116892 6164 test_util.cc:268] Using random seed: 955207737
I20240502 08:16:39.118579 6164 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:16:39.118659 6164 ts_itest-base.cc:113] --------------
I20240502 08:16:39.118714 6164 ts_itest-base.cc:114] 5 tablet servers
I20240502 08:16:39.118775 6164 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:16:39.118826 6164 ts_itest-base.cc:116] --------------
2024-05-02T08:16:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:16:39Z Disabled control of system clock
I20240502 08:16:39.137847 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.5.62:38477
--webserver_interface=127.6.5.62
--webserver_port=0
--builtin_ntp_servers=127.6.5.20:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.5.62:38477
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 08:16:39.433985 7036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:39.434259 7036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:39.434397 7036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:39.441829 7036 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:39.441983 7036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:39.442061 7036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:39.442152 7036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:39.442222 7036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:39.451376 7036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.5.62:38477
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.5.62:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.5.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:39.452628 7036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:39.454439 7036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:39.465194 7041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:39.465646 7044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:39.466135 7042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:39.466638 7036 server_base.cc:1034] running on GCE node
I20240502 08:16:39.467296 7036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:39.468621 7036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:39.469813 7036 hybrid_clock.cc:648] HybridClock initialized: now 1714637799469764 us; error 41 us; skew 500 ppm
I20240502 08:16:39.470121 7036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:39.472187 7036 webserver.cc:460] Webserver started at http://127.6.5.62:45801/ using document root <none> and password file <none>
I20240502 08:16:39.472837 7036 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:39.472942 7036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:39.473245 7036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:39.477557 7036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7aa751c8dca3494f99fbef6dff5f3a35"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-h7h6"
I20240502 08:16:39.478283 7036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7aa751c8dca3494f99fbef6dff5f3a35"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-h7h6"
I20240502 08:16:39.483040 7036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20240502 08:16:39.486084 7050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:39.487351 7036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:39.487506 7036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7aa751c8dca3494f99fbef6dff5f3a35"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-h7h6"
I20240502 08:16:39.487692 7036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:39.508625 7036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:39.509323 7036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:39.509572 7036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:39.525980 7036 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.62:38477
I20240502 08:16:39.526080 7101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.62:38477 every 8 connection(s)
I20240502 08:16:39.527189 7036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:16:39.529449 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 7036
I20240502 08:16:39.529649 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:16:39.530797 7102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.544100 7102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35: Bootstrap starting.
I20240502 08:16:39.546931 7102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.547905 7102 log.cc:824] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:39.550246 7102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35: No bootstrap required, opened a new log
I20240502 08:16:39.555989 7102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa751c8dca3494f99fbef6dff5f3a35" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 38477 } }
I20240502 08:16:39.556282 7102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.556376 7102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa751c8dca3494f99fbef6dff5f3a35, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.556875 7102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa751c8dca3494f99fbef6dff5f3a35" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 38477 } }
I20240502 08:16:39.557072 7102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:39.557181 7102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:39.557317 7102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:39.559232 7102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa751c8dca3494f99fbef6dff5f3a35" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 38477 } }
I20240502 08:16:39.559656 7102 leader_election.cc:302] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aa751c8dca3494f99fbef6dff5f3a35; no voters:
I20240502 08:16:39.560156 7102 leader_election.cc:288] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:39.560266 7107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:39.561226 7107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [term 1 LEADER]: Becoming Leader. State: Replica: 7aa751c8dca3494f99fbef6dff5f3a35, State: Running, Role: LEADER
I20240502 08:16:39.561677 7107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa751c8dca3494f99fbef6dff5f3a35" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 38477 } }
I20240502 08:16:39.561866 7102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:39.564255 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aa751c8dca3494f99fbef6dff5f3a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa751c8dca3494f99fbef6dff5f3a35" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 38477 } } }
I20240502 08:16:39.564291 7109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aa751c8dca3494f99fbef6dff5f3a35. Latest consensus state: current_term: 1 leader_uuid: "7aa751c8dca3494f99fbef6dff5f3a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa751c8dca3494f99fbef6dff5f3a35" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 38477 } } }
I20240502 08:16:39.564566 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:39.564594 7109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:39.566344 7114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:39.571043 7114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:39.578573 7114 catalog_manager.cc:1347] Generated new cluster ID: d64ccec2f14149ba92582b235ab13985
I20240502 08:16:39.578693 7114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:39.592015 7114 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:39.593183 7114 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:39.605887 7114 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7aa751c8dca3494f99fbef6dff5f3a35: Generated new TSK 0
I20240502 08:16:39.606652 7114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:39.615789 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.1:0
--local_ip_for_outbound_sockets=127.6.5.1
--webserver_interface=127.6.5.1
--webserver_port=0
--tserver_master_addrs=127.6.5.62:38477
--builtin_ntp_servers=127.6.5.20:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:16:39.938023 7126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:39.938277 7126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:39.938401 7126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:39.945458 7126 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:39.945570 7126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:39.945714 7126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.1
I20240502 08:16:39.954079 7126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.5.1
--webserver_port=0
--tserver_master_addrs=127.6.5.62:38477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.1
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:39.955224 7126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:39.956806 7126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:39.968120 7131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:39.968596 7132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:39.969019 7134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:39.969520 7126 server_base.cc:1034] running on GCE node
I20240502 08:16:39.970043 7126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:39.970896 7126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:39.972095 7126 hybrid_clock.cc:648] HybridClock initialized: now 1714637799972060 us; error 49 us; skew 500 ppm
I20240502 08:16:39.972380 7126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:39.974319 7126 webserver.cc:460] Webserver started at http://127.6.5.1:33585/ using document root <none> and password file <none>
I20240502 08:16:39.974992 7126 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:39.975098 7126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:39.975394 7126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:39.977710 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1a02777e03f74851b4ea284e9080d496"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-h7h6"
I20240502 08:16:39.978554 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1a02777e03f74851b4ea284e9080d496"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-h7h6"
I20240502 08:16:39.983309 7126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 08:16:39.986400 7140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:39.987592 7126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:16:39.987754 7126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1a02777e03f74851b4ea284e9080d496"
format_stamp: "Formatted at 2024-05-02 08:16:39 on dist-test-slave-h7h6"
I20240502 08:16:39.987929 7126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:40.001142 7126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:40.001827 7126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:40.002058 7126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:40.002933 7126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:40.004241 7126 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:40.004334 7126 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:40.004436 7126 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:40.004505 7126 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:40.031920 7126 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.1:36845
I20240502 08:16:40.031983 7252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.1:36845 every 8 connection(s)
I20240502 08:16:40.033105 7126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:16:40.043099 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 7126
I20240502 08:16:40.043275 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:16:40.044795 7253 heartbeater.cc:347] Connected to a master server at 127.6.5.62:38477
I20240502 08:16:40.045046 7253 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:40.045787 7253 heartbeater.cc:511] Master 127.6.5.62:38477 requested a full tablet report, sending...
I20240502 08:16:40.046413 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.2:0
--local_ip_for_outbound_sockets=127.6.5.2
--webserver_interface=127.6.5.2
--webserver_port=0
--tserver_master_addrs=127.6.5.62:38477
--builtin_ntp_servers=127.6.5.20:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:40.047713 7067 ts_manager.cc:194] Registered new tserver with Master: 1a02777e03f74851b4ea284e9080d496 (127.6.5.1:36845)
I20240502 08:16:40.049182 7067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.1:53263
W20240502 08:16:40.346767 7256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:40.347018 7256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:40.347151 7256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:40.356623 7256 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:40.356734 7256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:40.356884 7256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.2
I20240502 08:16:40.365294 7256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.5.2
--webserver_port=0
--tserver_master_addrs=127.6.5.62:38477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.2
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:40.366493 7256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:40.368188 7256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:40.379470 7262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:40.379899 7261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:40.380328 7264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:40.380872 7256 server_base.cc:1034] running on GCE node
I20240502 08:16:40.381371 7256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:40.382200 7256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:40.383383 7256 hybrid_clock.cc:648] HybridClock initialized: now 1714637800383347 us; error 35 us; skew 500 ppm
I20240502 08:16:40.383661 7256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:40.389757 7256 webserver.cc:460] Webserver started at http://127.6.5.2:35869/ using document root <none> and password file <none>
I20240502 08:16:40.390425 7256 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:40.390555 7256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:40.390868 7256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:40.393440 7256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a"
format_stamp: "Formatted at 2024-05-02 08:16:40 on dist-test-slave-h7h6"
I20240502 08:16:40.394127 7256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a"
format_stamp: "Formatted at 2024-05-02 08:16:40 on dist-test-slave-h7h6"
I20240502 08:16:40.398748 7256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20240502 08:16:40.401826 7270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:40.403095 7256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:40.403244 7256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a"
format_stamp: "Formatted at 2024-05-02 08:16:40 on dist-test-slave-h7h6"
I20240502 08:16:40.403417 7256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:40.420392 7256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:40.421048 7256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:40.421268 7256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:40.422044 7256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:40.423409 7256 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:40.423493 7256 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:40.423568 7256 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:40.423619 7256 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:40.451534 7256 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.2:33019
I20240502 08:16:40.451597 7382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.2:33019 every 8 connection(s)
I20240502 08:16:40.452694 7256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:16:40.460219 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 7256
I20240502 08:16:40.460420 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:16:40.463637 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.3:0
--local_ip_for_outbound_sockets=127.6.5.3
--webserver_interface=127.6.5.3
--webserver_port=0
--tserver_master_addrs=127.6.5.62:38477
--builtin_ntp_servers=127.6.5.20:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:40.464504 7383 heartbeater.cc:347] Connected to a master server at 127.6.5.62:38477
I20240502 08:16:40.464771 7383 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:40.465442 7383 heartbeater.cc:511] Master 127.6.5.62:38477 requested a full tablet report, sending...
I20240502 08:16:40.466676 7067 ts_manager.cc:194] Registered new tserver with Master: dc8c2b98dbbb4cef887e075fe00e8a2a (127.6.5.2:33019)
I20240502 08:16:40.467435 7067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.2:56325
W20240502 08:16:40.762163 7386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:40.762400 7386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:40.762526 7386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:40.769518 7386 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:40.769611 7386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:40.769748 7386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.3
I20240502 08:16:40.777983 7386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.5.3
--webserver_port=0
--tserver_master_addrs=127.6.5.62:38477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.3
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:40.779049 7386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:40.780566 7386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:40.791666 7391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:40.792125 7392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:40.792443 7394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:40.793030 7386 server_base.cc:1034] running on GCE node
I20240502 08:16:40.793536 7386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:40.794353 7386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:40.795550 7386 hybrid_clock.cc:648] HybridClock initialized: now 1714637800795516 us; error 57 us; skew 500 ppm
I20240502 08:16:40.795832 7386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:40.797741 7386 webserver.cc:460] Webserver started at http://127.6.5.3:41259/ using document root <none> and password file <none>
I20240502 08:16:40.798369 7386 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:40.798491 7386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:40.798796 7386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:40.801000 7386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c4cf3795cd4e4646ace57defa3a07ae1"
format_stamp: "Formatted at 2024-05-02 08:16:40 on dist-test-slave-h7h6"
I20240502 08:16:40.801680 7386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c4cf3795cd4e4646ace57defa3a07ae1"
format_stamp: "Formatted at 2024-05-02 08:16:40 on dist-test-slave-h7h6"
I20240502 08:16:40.806125 7386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20240502 08:16:40.809137 7400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:40.810261 7386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:40.810400 7386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c4cf3795cd4e4646ace57defa3a07ae1"
format_stamp: "Formatted at 2024-05-02 08:16:40 on dist-test-slave-h7h6"
I20240502 08:16:40.810628 7386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:40.833719 7386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:40.834424 7386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:40.834697 7386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:40.835525 7386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:40.836823 7386 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:40.836915 7386 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:40.837016 7386 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:40.837085 7386 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:40.864897 7386 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.3:35091
I20240502 08:16:40.864955 7512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.3:35091 every 8 connection(s)
I20240502 08:16:40.865980 7386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:16:40.868332 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 7386
I20240502 08:16:40.868511 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:16:40.871778 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.4:0
--local_ip_for_outbound_sockets=127.6.5.4
--webserver_interface=127.6.5.4
--webserver_port=0
--tserver_master_addrs=127.6.5.62:38477
--builtin_ntp_servers=127.6.5.20:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:40.878090 7513 heartbeater.cc:347] Connected to a master server at 127.6.5.62:38477
I20240502 08:16:40.878333 7513 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:40.878973 7513 heartbeater.cc:511] Master 127.6.5.62:38477 requested a full tablet report, sending...
I20240502 08:16:40.880208 7067 ts_manager.cc:194] Registered new tserver with Master: c4cf3795cd4e4646ace57defa3a07ae1 (127.6.5.3:35091)
I20240502 08:16:40.880964 7067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.3:52221
I20240502 08:16:41.052052 7253 heartbeater.cc:503] Master 127.6.5.62:38477 was elected leader, sending a full tablet report...
W20240502 08:16:41.171566 7516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:41.171830 7516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:41.171943 7516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:41.178925 7516 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:41.179037 7516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:41.179183 7516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.4
I20240502 08:16:41.187422 7516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.5.4
--webserver_port=0
--tserver_master_addrs=127.6.5.62:38477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.4
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:41.188522 7516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:41.190085 7516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:41.201593 7524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:41.202076 7522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:41.202533 7521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:41.203035 7516 server_base.cc:1034] running on GCE node
I20240502 08:16:41.203521 7516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:41.204325 7516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:41.205520 7516 hybrid_clock.cc:648] HybridClock initialized: now 1714637801205470 us; error 41 us; skew 500 ppm
I20240502 08:16:41.205801 7516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:41.207835 7516 webserver.cc:460] Webserver started at http://127.6.5.4:39345/ using document root <none> and password file <none>
I20240502 08:16:41.208505 7516 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:41.208612 7516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:41.208945 7516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:41.211385 7516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "94b388ad18154b13bb0271db0d798143"
format_stamp: "Formatted at 2024-05-02 08:16:41 on dist-test-slave-h7h6"
I20240502 08:16:41.212083 7516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "94b388ad18154b13bb0271db0d798143"
format_stamp: "Formatted at 2024-05-02 08:16:41 on dist-test-slave-h7h6"
I20240502 08:16:41.216766 7516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:16:41.220055 7530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:41.221241 7516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20240502 08:16:41.221380 7516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "94b388ad18154b13bb0271db0d798143"
format_stamp: "Formatted at 2024-05-02 08:16:41 on dist-test-slave-h7h6"
I20240502 08:16:41.221556 7516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:41.235332 7516 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:41.236028 7516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:41.236274 7516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:41.237125 7516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:41.238430 7516 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:41.238548 7516 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:41.238654 7516 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:41.238732 7516 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:41.267211 7516 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.4:45757
I20240502 08:16:41.267292 7642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.4:45757 every 8 connection(s)
I20240502 08:16:41.268390 7516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:16:41.276436 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 7516
I20240502 08:16:41.276613 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:16:41.279774 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.5:0
--local_ip_for_outbound_sockets=127.6.5.5
--webserver_interface=127.6.5.5
--webserver_port=0
--tserver_master_addrs=127.6.5.62:38477
--builtin_ntp_servers=127.6.5.20:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:41.280022 7643 heartbeater.cc:347] Connected to a master server at 127.6.5.62:38477
I20240502 08:16:41.280263 7643 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:41.280913 7643 heartbeater.cc:511] Master 127.6.5.62:38477 requested a full tablet report, sending...
I20240502 08:16:41.282088 7067 ts_manager.cc:194] Registered new tserver with Master: 94b388ad18154b13bb0271db0d798143 (127.6.5.4:45757)
I20240502 08:16:41.282876 7067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.4:34445
I20240502 08:16:41.470041 7383 heartbeater.cc:503] Master 127.6.5.62:38477 was elected leader, sending a full tablet report...
W20240502 08:16:41.577517 7646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:41.577764 7646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:41.577843 7646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:41.584795 7646 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:41.584892 7646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:41.585011 7646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.5
I20240502 08:16:41.593402 7646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.5.5
--webserver_port=0
--tserver_master_addrs=127.6.5.62:38477
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.5
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:41.594527 7646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:41.596092 7646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:41.607244 7652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:41.607841 7654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:41.608275 7651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:41.608788 7646 server_base.cc:1034] running on GCE node
I20240502 08:16:41.609330 7646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:41.610183 7646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:41.611379 7646 hybrid_clock.cc:648] HybridClock initialized: now 1714637801611326 us; error 43 us; skew 500 ppm
I20240502 08:16:41.611696 7646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:41.613771 7646 webserver.cc:460] Webserver started at http://127.6.5.5:38981/ using document root <none> and password file <none>
I20240502 08:16:41.614493 7646 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:41.614598 7646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:41.614893 7646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:41.617274 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "18173b5c6ecd4a77ada93f82ae85a1a8"
format_stamp: "Formatted at 2024-05-02 08:16:41 on dist-test-slave-h7h6"
I20240502 08:16:41.617942 7646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "18173b5c6ecd4a77ada93f82ae85a1a8"
format_stamp: "Formatted at 2024-05-02 08:16:41 on dist-test-slave-h7h6"
I20240502 08:16:41.622483 7646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 08:16:41.625536 7660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:41.626739 7646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 08:16:41.626875 7646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "18173b5c6ecd4a77ada93f82ae85a1a8"
format_stamp: "Formatted at 2024-05-02 08:16:41 on dist-test-slave-h7h6"
I20240502 08:16:41.627060 7646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:41.639779 7646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:41.640444 7646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:41.640671 7646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:41.641454 7646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:41.642783 7646 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:41.642877 7646 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:41.642983 7646 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:41.643049 7646 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:41.670666 7646 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.5:33217
I20240502 08:16:41.670720 7772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.5:33217 every 8 connection(s)
I20240502 08:16:41.671798 7646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:16:41.673149 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 7646
I20240502 08:16:41.673301 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:16:41.683359 7773 heartbeater.cc:347] Connected to a master server at 127.6.5.62:38477
I20240502 08:16:41.683606 7773 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:41.684242 7773 heartbeater.cc:511] Master 127.6.5.62:38477 requested a full tablet report, sending...
I20240502 08:16:41.685442 7067 ts_manager.cc:194] Registered new tserver with Master: 18173b5c6ecd4a77ada93f82ae85a1a8 (127.6.5.5:33217)
I20240502 08:16:41.686180 7067 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.5:37073
I20240502 08:16:41.688897 6164 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:16:41.703133 7067 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49954:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 08:16:41.746670 7448 tablet_service.cc:1467] Processing CreateTablet for tablet d2a8bd888c0b4cbd99e9515c5130b4fc (DEFAULT_TABLE table=TestTable [id=59a2f1be2ac740c6883f6a413ea5845c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.747650 7708 tablet_service.cc:1467] Processing CreateTablet for tablet d2a8bd888c0b4cbd99e9515c5130b4fc (DEFAULT_TABLE table=TestTable [id=59a2f1be2ac740c6883f6a413ea5845c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.748188 7448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a8bd888c0b4cbd99e9515c5130b4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.748912 7708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a8bd888c0b4cbd99e9515c5130b4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.748493 7318 tablet_service.cc:1467] Processing CreateTablet for tablet d2a8bd888c0b4cbd99e9515c5130b4fc (DEFAULT_TABLE table=TestTable [id=59a2f1be2ac740c6883f6a413ea5845c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.749783 7318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a8bd888c0b4cbd99e9515c5130b4fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.758808 7791 tablet_bootstrap.cc:492] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8: Bootstrap starting.
I20240502 08:16:41.761503 7791 tablet_bootstrap.cc:654] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.762157 7792 tablet_bootstrap.cc:492] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1: Bootstrap starting.
I20240502 08:16:41.762981 7791 log.cc:824] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:41.763583 7793 tablet_bootstrap.cc:492] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a: Bootstrap starting.
I20240502 08:16:41.765527 7792 tablet_bootstrap.cc:654] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.766773 7792 log.cc:824] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:41.766878 7793 tablet_bootstrap.cc:654] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.768071 7793 log.cc:824] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:41.769162 7791 tablet_bootstrap.cc:492] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8: No bootstrap required, opened a new log
I20240502 08:16:41.769423 7791 ts_tablet_manager.cc:1367] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 08:16:41.769907 7792 tablet_bootstrap.cc:492] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1: No bootstrap required, opened a new log
I20240502 08:16:41.770155 7792 ts_tablet_manager.cc:1367] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 08:16:41.770373 7793 tablet_bootstrap.cc:492] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a: No bootstrap required, opened a new log
I20240502 08:16:41.770633 7793 ts_tablet_manager.cc:1367] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240502 08:16:41.776222 7793 raft_consensus.cc:348] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 33019 } } peers { permanent_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 35091 } } peers { permanent_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 33217 } }
I20240502 08:16:41.776574 7793 raft_consensus.cc:374] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.776681 7793 raft_consensus.cc:729] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8c2b98dbbb4cef887e075fe00e8a2a, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.777583 7793 consensus_queue.cc:260] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 33019 } } peers { permanent_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 35091 } } peers { permanent_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 33217 } }
I20240502 08:16:41.777577 7791 raft_consensus.cc:348] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 33019 } } peers { permanent_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 35091 } } peers { permanent_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 33217 } }
I20240502 08:16:41.777969 7791 raft_consensus.cc:374] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.778074 7791 raft_consensus.cc:729] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18173b5c6ecd4a77ada93f82ae85a1a8, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.778208 7792 raft_consensus.cc:348] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 33019 } } peers { permanent_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 35091 } } peers { permanent_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 33217 } }
I20240502 08:16:41.778589 7792 raft_consensus.cc:374] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.778694 7792 raft_consensus.cc:729] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cf3795cd4e4646ace57defa3a07ae1, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.778815 7791 consensus_queue.cc:260] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 33019 } } peers { permanent_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 35091 } } peers { permanent_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 33217 } }
I20240502 08:16:41.779299 7792 consensus_queue.cc:260] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 33019 } } peers { permanent_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 35091 } } peers { permanent_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 33217 } }
I20240502 08:16:41.779798 7793 ts_tablet_manager.cc:1398] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240502 08:16:41.781360 7513 heartbeater.cc:503] Master 127.6.5.62:38477 was elected leader, sending a full tablet report...
I20240502 08:16:41.781515 7792 ts_tablet_manager.cc:1398] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20240502 08:16:41.784034 7773 heartbeater.cc:503] Master 127.6.5.62:38477 was elected leader, sending a full tablet report...
I20240502 08:16:41.784543 7791 ts_tablet_manager.cc:1398] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20240502 08:16:41.848644 7799 raft_consensus.cc:482] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:41.848853 7799 raft_consensus.cc:504] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 33019 } } peers { permanent_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 35091 } } peers { permanent_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 33217 } }
I20240502 08:16:41.850684 7799 leader_election.cc:288] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc8c2b98dbbb4cef887e075fe00e8a2a (127.6.5.2:33019), 18173b5c6ecd4a77ada93f82ae85a1a8 (127.6.5.5:33217)
I20240502 08:16:41.858798 7728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a8bd888c0b4cbd99e9515c5130b4fc" candidate_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8" is_pre_election: true
I20240502 08:16:41.858827 7338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a8bd888c0b4cbd99e9515c5130b4fc" candidate_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a" is_pre_election: true
I20240502 08:16:41.859220 7728 raft_consensus.cc:2454] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4cf3795cd4e4646ace57defa3a07ae1 in term 0.
I20240502 08:16:41.859233 7338 raft_consensus.cc:2454] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4cf3795cd4e4646ace57defa3a07ae1 in term 0.
I20240502 08:16:41.859748 7402 leader_election.cc:302] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4cf3795cd4e4646ace57defa3a07ae1, dc8c2b98dbbb4cef887e075fe00e8a2a; no voters:
I20240502 08:16:41.860080 7799 raft_consensus.cc:2781] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:41.860198 7799 raft_consensus.cc:482] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:41.860287 7799 raft_consensus.cc:3037] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:41.862195 7799 raft_consensus.cc:504] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 33019 } } peers { permanent_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 35091 } } peers { permanent_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 33217 } }
I20240502 08:16:41.862757 7799 leader_election.cc:288] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [CANDIDATE]: Term 1 election: Requested vote from peers dc8c2b98dbbb4cef887e075fe00e8a2a (127.6.5.2:33019), 18173b5c6ecd4a77ada93f82ae85a1a8 (127.6.5.5:33217)
I20240502 08:16:41.863098 7338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a8bd888c0b4cbd99e9515c5130b4fc" candidate_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a"
I20240502 08:16:41.863183 7728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a8bd888c0b4cbd99e9515c5130b4fc" candidate_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8"
I20240502 08:16:41.863257 7338 raft_consensus.cc:3037] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:41.863461 7728 raft_consensus.cc:3037] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:41.865386 7338 raft_consensus.cc:2454] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4cf3795cd4e4646ace57defa3a07ae1 in term 1.
I20240502 08:16:41.865720 7402 leader_election.cc:302] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4cf3795cd4e4646ace57defa3a07ae1, dc8c2b98dbbb4cef887e075fe00e8a2a; no voters:
I20240502 08:16:41.865974 7799 raft_consensus.cc:2781] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:41.866154 7728 raft_consensus.cc:2454] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4cf3795cd4e4646ace57defa3a07ae1 in term 1.
I20240502 08:16:41.866451 7799 raft_consensus.cc:686] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [term 1 LEADER]: Becoming Leader. State: Replica: c4cf3795cd4e4646ace57defa3a07ae1, State: Running, Role: LEADER
I20240502 08:16:41.866940 7799 consensus_queue.cc:237] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 33019 } } peers { permanent_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 35091 } } peers { permanent_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 33217 } }
W20240502 08:16:41.868386 7514 tablet.cc:2369] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:41.872112 7067 catalog_manager.cc:5484] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4cf3795cd4e4646ace57defa3a07ae1 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 33019 } } peers { permanent_uuid: "c4cf3795cd4e4646ace57defa3a07ae1" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 35091 } } peers { permanent_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 33217 } } }
I20240502 08:16:41.910632 6164 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:16:41.912534 6164 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1a02777e03f74851b4ea284e9080d496 to finish bootstrapping
I20240502 08:16:41.921597 6164 ts_itest-base.cc:233] Waiting for 1 tablets on tserver dc8c2b98dbbb4cef887e075fe00e8a2a to finish bootstrapping
W20240502 08:16:41.923270 7774 tablet.cc:2369] T d2a8bd888c0b4cbd99e9515c5130b4fc P 18173b5c6ecd4a77ada93f82ae85a1a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:41.928356 6164 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c4cf3795cd4e4646ace57defa3a07ae1 to finish bootstrapping
I20240502 08:16:41.935041 6164 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 94b388ad18154b13bb0271db0d798143 to finish bootstrapping
I20240502 08:16:41.942642 6164 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 18173b5c6ecd4a77ada93f82ae85a1a8 to finish bootstrapping
I20240502 08:16:41.949283 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 7126
I20240502 08:16:41.950122 7248 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:16:41.955111 7384 tablet.cc:2369] T d2a8bd888c0b4cbd99e9515c5130b4fc P dc8c2b98dbbb4cef887e075fe00e8a2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:42.029467 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 7126
I20240502 08:16:42.285104 7643 heartbeater.cc:503] Master 127.6.5.62:38477 was elected leader, sending a full tablet report...
I20240502 08:16:42.354898 7799 consensus_queue.cc:1035] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc8c2b98dbbb4cef887e075fe00e8a2a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 33019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:42.363057 7802 consensus_queue.cc:1035] T d2a8bd888c0b4cbd99e9515c5130b4fc P c4cf3795cd4e4646ace57defa3a07ae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18173b5c6ecd4a77ada93f82ae85a1a8" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 33217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:42.489609 7708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:42.490988 7578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:42.493258 7448 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:42.498610 7318 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:42.617412 7708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:42.628616 7578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:42.632330 7448 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:42.639523 7318 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:42.789628 7578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:42.790222 7708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:42.797108 7448 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:42.807969 7318 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:43.353339 7578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:43.354593 7708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:43.360005 7318 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:43.362813 7448 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:43.494822 7708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:43.494875 7578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:43.499940 7448 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:43.501137 7318 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:43.654357 7708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:43.656991 7578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:43.664130 7318 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:43.689715 7448 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:43.816383 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 7256
I20240502 08:16:43.817668 7377 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:43.912731 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 7256
I20240502 08:16:43.927337 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 7386
I20240502 08:16:43.928298 7507 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:44.020838 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 7386
I20240502 08:16:44.035481 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 7516
I20240502 08:16:44.036394 7638 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:44.123317 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 7516
I20240502 08:16:44.136639 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 7646
I20240502 08:16:44.137545 7767 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:44.229974 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 7646
I20240502 08:16:44.243916 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 7036
I20240502 08:16:44.244912 7097 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:44.358089 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 7036
2024-05-02T08:16:44Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (5279 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5279 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240502 08:16:44.396306 6164 test_util.cc:268] Using random seed: 960487162
I20240502 08:16:44.398201 6164 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:16:44.398273 6164 ts_itest-base.cc:113] --------------
I20240502 08:16:44.398332 6164 ts_itest-base.cc:114] 7 tablet servers
I20240502 08:16:44.398386 6164 ts_itest-base.cc:115] 1 replicas per TS
I20240502 08:16:44.398453 6164 ts_itest-base.cc:116] --------------
2024-05-02T08:16:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:16:44Z Disabled control of system clock
I20240502 08:16:44.418298 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.5.62:44177
--webserver_interface=127.6.5.62
--webserver_port=0
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.5.62:44177
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 08:16:44.713340 8013 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 08:16:44.713654 8013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:44.713757 8013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:44.713855 8013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:44.721060 8013 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:44.721177 8013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:44.721268 8013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:44.721328 8013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:44.721415 8013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:44.730160 8013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.5.62:44177
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.5.62:44177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.5.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:44.731369 8013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:44.733000 8013 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:44.743432 8018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:44.743875 8019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:44.744239 8021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:44.744776 8013 server_base.cc:1034] running on GCE node
I20240502 08:16:44.745540 8013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:44.746868 8013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:44.748106 8013 hybrid_clock.cc:648] HybridClock initialized: now 1714637804748042 us; error 35 us; skew 500 ppm
I20240502 08:16:44.748425 8013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:44.750624 8013 webserver.cc:460] Webserver started at http://127.6.5.62:37255/ using document root <none> and password file <none>
I20240502 08:16:44.751267 8013 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:44.751389 8013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:44.751657 8013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:44.755671 8013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1e7a91bc1fb549619d016b96f81b6ff5"
format_stamp: "Formatted at 2024-05-02 08:16:44 on dist-test-slave-h7h6"
I20240502 08:16:44.756350 8013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1e7a91bc1fb549619d016b96f81b6ff5"
format_stamp: "Formatted at 2024-05-02 08:16:44 on dist-test-slave-h7h6"
I20240502 08:16:44.761111 8013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:16:44.764271 8027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:44.765455 8013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 08:16:44.765599 8013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1e7a91bc1fb549619d016b96f81b6ff5"
format_stamp: "Formatted at 2024-05-02 08:16:44 on dist-test-slave-h7h6"
I20240502 08:16:44.765753 8013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:44.776948 8013 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:44.777611 8013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:44.777856 8013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:44.793391 8013 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.62:44177
I20240502 08:16:44.793429 8078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.62:44177 every 8 connection(s)
I20240502 08:16:44.794575 8013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:16:44.798032 8079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.800432 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 8013
I20240502 08:16:44.800623 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:16:44.808436 8079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5: Bootstrap starting.
I20240502 08:16:44.811902 8079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.813217 8079 log.cc:824] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:44.815596 8079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5: No bootstrap required, opened a new log
I20240502 08:16:44.821487 8079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e7a91bc1fb549619d016b96f81b6ff5" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 44177 } }
I20240502 08:16:44.821766 8079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.821843 8079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7a91bc1fb549619d016b96f81b6ff5, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.822307 8079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e7a91bc1fb549619d016b96f81b6ff5" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 44177 } }
I20240502 08:16:44.822503 8079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:44.822600 8079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:44.822712 8079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.824586 8079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e7a91bc1fb549619d016b96f81b6ff5" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 44177 } }
I20240502 08:16:44.825031 8079 leader_election.cc:302] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e7a91bc1fb549619d016b96f81b6ff5; no voters:
I20240502 08:16:44.825623 8079 leader_election.cc:288] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:44.825742 8084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:44.826292 8084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 1e7a91bc1fb549619d016b96f81b6ff5, State: Running, Role: LEADER
I20240502 08:16:44.826874 8084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e7a91bc1fb549619d016b96f81b6ff5" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 44177 } }
I20240502 08:16:44.827684 8079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:44.829322 8086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e7a91bc1fb549619d016b96f81b6ff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e7a91bc1fb549619d016b96f81b6ff5" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 44177 } } }
I20240502 08:16:44.829298 8085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e7a91bc1fb549619d016b96f81b6ff5. Latest consensus state: current_term: 1 leader_uuid: "1e7a91bc1fb549619d016b96f81b6ff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e7a91bc1fb549619d016b96f81b6ff5" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 44177 } } }
I20240502 08:16:44.829726 8085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:44.829730 8086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:44.832616 8089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:44.837170 8089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:44.844468 8089 catalog_manager.cc:1347] Generated new cluster ID: d04f31df4b69440cac9049dae7077a2f
I20240502 08:16:44.844568 8089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:44.872726 8089 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:44.874246 8089 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:44.882575 8089 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1e7a91bc1fb549619d016b96f81b6ff5: Generated new TSK 0
I20240502 08:16:44.883421 8089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:44.889132 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.1:0
--local_ip_for_outbound_sockets=127.6.5.1
--webserver_interface=127.6.5.1
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:16:45.190873 8103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:45.191135 8103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:45.191269 8103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:45.198513 8103 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:45.198628 8103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:45.198774 8103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.1
I20240502 08:16:45.207286 8103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.5.1
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.1
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:45.208545 8103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:45.210366 8103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:45.221606 8109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:45.222028 8108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:45.222438 8111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:45.222946 8103 server_base.cc:1034] running on GCE node
I20240502 08:16:45.223412 8103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:45.224195 8103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:45.225382 8103 hybrid_clock.cc:648] HybridClock initialized: now 1714637805225343 us; error 29 us; skew 500 ppm
I20240502 08:16:45.225639 8103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:45.227728 8103 webserver.cc:460] Webserver started at http://127.6.5.1:34567/ using document root <none> and password file <none>
I20240502 08:16:45.228351 8103 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:45.228440 8103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:45.228724 8103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:45.231072 8103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b90db359bc744f2b82b69d97627fa7b9"
format_stamp: "Formatted at 2024-05-02 08:16:45 on dist-test-slave-h7h6"
I20240502 08:16:45.231719 8103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b90db359bc744f2b82b69d97627fa7b9"
format_stamp: "Formatted at 2024-05-02 08:16:45 on dist-test-slave-h7h6"
I20240502 08:16:45.236152 8103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240502 08:16:45.239037 8117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:45.240193 8103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:45.240315 8103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b90db359bc744f2b82b69d97627fa7b9"
format_stamp: "Formatted at 2024-05-02 08:16:45 on dist-test-slave-h7h6"
I20240502 08:16:45.240476 8103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:45.261734 8103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:45.262377 8103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:45.262607 8103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:45.263366 8103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:45.264715 8103 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:45.264796 8103 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:45.264868 8103 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:45.264918 8103 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:45.291920 8103 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.1:35591
I20240502 08:16:45.291980 8229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.1:35591 every 8 connection(s)
I20240502 08:16:45.293042 8103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:16:45.294520 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 8103
I20240502 08:16:45.294728 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:16:45.297964 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.2:0
--local_ip_for_outbound_sockets=127.6.5.2
--webserver_interface=127.6.5.2
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:45.305246 8230 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:16:45.305490 8230 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:45.306128 8230 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:16:45.308164 8044 ts_manager.cc:194] Registered new tserver with Master: b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:16:45.310261 8044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.1:59653
W20240502 08:16:45.596758 8233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:45.597039 8233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:45.597141 8233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:45.604068 8233 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:45.604162 8233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:45.604272 8233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.2
I20240502 08:16:45.612541 8233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.5.2
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.2
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:45.613655 8233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:45.615271 8233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:45.626423 8238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:45.626926 8239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:45.627306 8241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:45.627790 8233 server_base.cc:1034] running on GCE node
I20240502 08:16:45.628324 8233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:45.629176 8233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:45.630359 8233 hybrid_clock.cc:648] HybridClock initialized: now 1714637805630310 us; error 40 us; skew 500 ppm
I20240502 08:16:45.630657 8233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:45.637400 8233 webserver.cc:460] Webserver started at http://127.6.5.2:34583/ using document root <none> and password file <none>
I20240502 08:16:45.638090 8233 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:45.638204 8233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:45.638532 8233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:45.640955 8233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "89ae15f3455446339957f9559f3857da"
format_stamp: "Formatted at 2024-05-02 08:16:45 on dist-test-slave-h7h6"
I20240502 08:16:45.641652 8233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "89ae15f3455446339957f9559f3857da"
format_stamp: "Formatted at 2024-05-02 08:16:45 on dist-test-slave-h7h6"
I20240502 08:16:45.646345 8233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 08:16:45.649416 8247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:45.650624 8233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:45.650761 8233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "89ae15f3455446339957f9559f3857da"
format_stamp: "Formatted at 2024-05-02 08:16:45 on dist-test-slave-h7h6"
I20240502 08:16:45.650936 8233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:45.663457 8233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:45.664134 8233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:45.664366 8233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:45.665143 8233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:45.666404 8233 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:45.666517 8233 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:45.666618 8233 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:45.666685 8233 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:45.693590 8233 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.2:44539
I20240502 08:16:45.693646 8359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.2:44539 every 8 connection(s)
I20240502 08:16:45.694749 8233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:16:45.703243 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 8233
I20240502 08:16:45.703423 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:16:45.706534 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.3:0
--local_ip_for_outbound_sockets=127.6.5.3
--webserver_interface=127.6.5.3
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:45.706712 8360 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:16:45.706982 8360 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:45.707826 8360 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:16:45.709141 8044 ts_manager.cc:194] Registered new tserver with Master: 89ae15f3455446339957f9559f3857da (127.6.5.2:44539)
I20240502 08:16:45.709864 8044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.2:35773
W20240502 08:16:46.004734 8363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:46.004987 8363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:46.005110 8363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:46.012126 8363 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:46.012228 8363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:46.012377 8363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.3
I20240502 08:16:46.020746 8363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.5.3
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.3
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:46.021852 8363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:46.023444 8363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:46.034741 8368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:46.035176 8371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:46.035583 8369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.036103 8363 server_base.cc:1034] running on GCE node
I20240502 08:16:46.036623 8363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:46.037429 8363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:46.038619 8363 hybrid_clock.cc:648] HybridClock initialized: now 1714637806038580 us; error 35 us; skew 500 ppm
I20240502 08:16:46.038905 8363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:46.040905 8363 webserver.cc:460] Webserver started at http://127.6.5.3:33467/ using document root <none> and password file <none>
I20240502 08:16:46.041612 8363 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:46.041723 8363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:46.042017 8363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:46.044533 8363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "00b1227597554ea18f7e125de6d2760e"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-h7h6"
I20240502 08:16:46.045271 8363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "00b1227597554ea18f7e125de6d2760e"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-h7h6"
I20240502 08:16:46.049865 8363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 08:16:46.052963 8377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.054311 8363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:46.054479 8363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "00b1227597554ea18f7e125de6d2760e"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-h7h6"
I20240502 08:16:46.054680 8363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:46.122915 8363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:46.123776 8363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:46.124032 8363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:46.124956 8363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:46.126379 8363 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:46.126487 8363 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.126588 8363 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:46.126644 8363 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.155457 8363 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.3:43567
I20240502 08:16:46.155510 8489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.3:43567 every 8 connection(s)
I20240502 08:16:46.156786 8363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:16:46.161670 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 8363
I20240502 08:16:46.161899 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:16:46.165151 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.4:0
--local_ip_for_outbound_sockets=127.6.5.4
--webserver_interface=127.6.5.4
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:46.170498 8490 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:16:46.170835 8490 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:46.171547 8490 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:16:46.172856 8044 ts_manager.cc:194] Registered new tserver with Master: 00b1227597554ea18f7e125de6d2760e (127.6.5.3:43567)
I20240502 08:16:46.173667 8044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.3:47869
I20240502 08:16:46.312875 8230 heartbeater.cc:503] Master 127.6.5.62:44177 was elected leader, sending a full tablet report...
W20240502 08:16:46.467178 8493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:46.467439 8493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:46.467567 8493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:46.474705 8493 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:46.474828 8493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:46.474988 8493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.4
I20240502 08:16:46.483371 8493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.5.4
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.4
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:46.484535 8493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:46.486168 8493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:46.497398 8498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:46.497836 8499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:46.498276 8501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.498764 8493 server_base.cc:1034] running on GCE node
I20240502 08:16:46.499270 8493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:46.500046 8493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:46.501224 8493 hybrid_clock.cc:648] HybridClock initialized: now 1714637806501179 us; error 32 us; skew 500 ppm
I20240502 08:16:46.501500 8493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:46.503470 8493 webserver.cc:460] Webserver started at http://127.6.5.4:35027/ using document root <none> and password file <none>
I20240502 08:16:46.504107 8493 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:46.504215 8493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:46.504492 8493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:46.506855 8493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b58bd494983a4a2885cda3387b1be902"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-h7h6"
I20240502 08:16:46.507546 8493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b58bd494983a4a2885cda3387b1be902"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-h7h6"
I20240502 08:16:46.512331 8493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:16:46.515273 8507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.516408 8493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:46.516548 8493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b58bd494983a4a2885cda3387b1be902"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-h7h6"
I20240502 08:16:46.516717 8493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:46.534257 8493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:46.534962 8493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:46.535205 8493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:46.535993 8493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:46.537276 8493 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:46.537369 8493 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.537470 8493 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:46.537536 8493 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.566042 8493 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.4:35529
I20240502 08:16:46.566098 8619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.4:35529 every 8 connection(s)
I20240502 08:16:46.567176 8493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:16:46.570226 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 8493
I20240502 08:16:46.570385 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:16:46.573751 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.5:0
--local_ip_for_outbound_sockets=127.6.5.5
--webserver_interface=127.6.5.5
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:46.579480 8620 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:16:46.579761 8620 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:46.580515 8620 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:16:46.581816 8044 ts_manager.cc:194] Registered new tserver with Master: b58bd494983a4a2885cda3387b1be902 (127.6.5.4:35529)
I20240502 08:16:46.582592 8044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.4:43327
I20240502 08:16:46.712391 8360 heartbeater.cc:503] Master 127.6.5.62:44177 was elected leader, sending a full tablet report...
W20240502 08:16:46.873481 8623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:46.873716 8623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:46.873805 8623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:46.880651 8623 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:46.880745 8623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:46.880853 8623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.5
I20240502 08:16:46.888978 8623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.5.5
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.5
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:46.889988 8623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:46.891499 8623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:46.902599 8628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:46.903052 8629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:46.903553 8631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:46.904120 8623 server_base.cc:1034] running on GCE node
I20240502 08:16:46.904610 8623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:46.905395 8623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:46.906543 8623 hybrid_clock.cc:648] HybridClock initialized: now 1714637806906510 us; error 24 us; skew 500 ppm
I20240502 08:16:46.906805 8623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:46.908921 8623 webserver.cc:460] Webserver started at http://127.6.5.5:42903/ using document root <none> and password file <none>
I20240502 08:16:46.909713 8623 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:46.909839 8623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:46.910151 8623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:46.912853 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c4e97cd51e9949339f937b6be4cefebb"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-h7h6"
I20240502 08:16:46.913542 8623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c4e97cd51e9949339f937b6be4cefebb"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-h7h6"
I20240502 08:16:46.918092 8623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20240502 08:16:46.921236 8637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.922438 8623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:46.922606 8623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c4e97cd51e9949339f937b6be4cefebb"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-h7h6"
I20240502 08:16:46.922780 8623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:46.949697 8623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:46.950361 8623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:46.950624 8623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:46.951462 8623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:46.952760 8623 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:46.952843 8623 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.952939 8623 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:46.953008 8623 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:46.985672 8623 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.5:34875
I20240502 08:16:46.985724 8749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.5:34875 every 8 connection(s)
I20240502 08:16:46.986884 8623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:16:46.989614 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 8623
I20240502 08:16:46.989789 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:16:46.992848 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.6:0
--local_ip_for_outbound_sockets=127.6.5.6
--webserver_interface=127.6.5.6
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:46.999558 8750 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:16:46.999840 8750 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:47.000584 8750 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:16:47.001866 8044 ts_manager.cc:194] Registered new tserver with Master: c4e97cd51e9949339f937b6be4cefebb (127.6.5.5:34875)
I20240502 08:16:47.002630 8044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.5:45173
I20240502 08:16:47.175951 8490 heartbeater.cc:503] Master 127.6.5.62:44177 was elected leader, sending a full tablet report...
W20240502 08:16:47.293462 8753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:47.293711 8753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:47.293833 8753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:47.300804 8753 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:47.300920 8753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:47.301059 8753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.6
I20240502 08:16:47.309249 8753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.5.6
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.6
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:47.310321 8753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:47.311910 8753 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:47.323103 8758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:47.323562 8759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:47.323949 8761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:47.324440 8753 server_base.cc:1034] running on GCE node
I20240502 08:16:47.324910 8753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:47.325680 8753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:47.326869 8753 hybrid_clock.cc:648] HybridClock initialized: now 1714637807326763 us; error 96 us; skew 500 ppm
I20240502 08:16:47.327152 8753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:47.329223 8753 webserver.cc:460] Webserver started at http://127.6.5.6:39643/ using document root <none> and password file <none>
I20240502 08:16:47.329872 8753 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:47.329975 8753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:47.330224 8753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:47.332624 8753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "646203131e6c42a2a2e631374efbe5ee"
format_stamp: "Formatted at 2024-05-02 08:16:47 on dist-test-slave-h7h6"
I20240502 08:16:47.333469 8753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "646203131e6c42a2a2e631374efbe5ee"
format_stamp: "Formatted at 2024-05-02 08:16:47 on dist-test-slave-h7h6"
I20240502 08:16:47.338539 8753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20240502 08:16:47.341696 8767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:47.342975 8753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:47.343147 8753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "646203131e6c42a2a2e631374efbe5ee"
format_stamp: "Formatted at 2024-05-02 08:16:47 on dist-test-slave-h7h6"
I20240502 08:16:47.343418 8753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:47.363454 8753 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:47.364218 8753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:47.364440 8753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:47.365240 8753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:47.366564 8753 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:47.366647 8753 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:47.366727 8753 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:47.366780 8753 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:47.394573 8753 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.6:34035
I20240502 08:16:47.394623 8879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.6:34035 every 8 connection(s)
I20240502 08:16:47.395726 8753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:16:47.397964 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 8753
I20240502 08:16:47.398154 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 08:16:47.401016 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.7:0
--local_ip_for_outbound_sockets=127.6.5.7
--webserver_interface=127.6.5.7
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:47.407801 8880 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:16:47.408054 8880 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:47.408676 8880 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:16:47.409832 8044 ts_manager.cc:194] Registered new tserver with Master: 646203131e6c42a2a2e631374efbe5ee (127.6.5.6:34035)
I20240502 08:16:47.410617 8044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.6:37319
I20240502 08:16:47.585178 8620 heartbeater.cc:503] Master 127.6.5.62:44177 was elected leader, sending a full tablet report...
W20240502 08:16:47.717139 8883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:47.717393 8883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:47.717492 8883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:47.724578 8883 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:47.724679 8883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:47.724792 8883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.7
I20240502 08:16:47.733268 8883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.5.7
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.7
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:47.734440 8883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:47.735998 8883 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:47.747218 8888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:47.747711 8889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:47.748174 8891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:47.748673 8883 server_base.cc:1034] running on GCE node
I20240502 08:16:47.749198 8883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:47.749966 8883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:47.751152 8883 hybrid_clock.cc:648] HybridClock initialized: now 1714637807751107 us; error 37 us; skew 500 ppm
I20240502 08:16:47.751426 8883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:47.753356 8883 webserver.cc:460] Webserver started at http://127.6.5.7:36201/ using document root <none> and password file <none>
I20240502 08:16:47.754009 8883 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:47.754118 8883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:47.754482 8883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:47.756863 8883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "6e2d5eaefade459c8e1fce0db41f2aca"
format_stamp: "Formatted at 2024-05-02 08:16:47 on dist-test-slave-h7h6"
I20240502 08:16:47.757557 8883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "6e2d5eaefade459c8e1fce0db41f2aca"
format_stamp: "Formatted at 2024-05-02 08:16:47 on dist-test-slave-h7h6"
I20240502 08:16:47.761991 8883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20240502 08:16:47.765067 8897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:47.766207 8883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:47.766350 8883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6e2d5eaefade459c8e1fce0db41f2aca"
format_stamp: "Formatted at 2024-05-02 08:16:47 on dist-test-slave-h7h6"
I20240502 08:16:47.766559 8883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:47.785264 8883 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:47.785952 8883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:47.786185 8883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:47.787025 8883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:47.788312 8883 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:47.788404 8883 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:47.788506 8883 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:47.788573 8883 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:47.817698 8883 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.7:39789
I20240502 08:16:47.817756 9009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.7:39789 every 8 connection(s)
I20240502 08:16:47.818882 8883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:16:47.825197 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 8883
I20240502 08:16:47.825361 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 08:16:47.831053 9010 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:16:47.831305 9010 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:47.831931 9010 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:16:47.833149 8044 ts_manager.cc:194] Registered new tserver with Master: 6e2d5eaefade459c8e1fce0db41f2aca (127.6.5.7:39789)
I20240502 08:16:47.833904 8044 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.7:57443
I20240502 08:16:47.841214 6164 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 08:16:47.855821 8044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42606:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 08:16:47.880021 8555 tablet_service.cc:1467] Processing CreateTablet for tablet da122388bec1424bb9e07919dd2d3cf0 (DEFAULT_TABLE table=TestTable [id=e9c38538183a456c80008c1c05fc7610]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.881204 8555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da122388bec1424bb9e07919dd2d3cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.890209 9024 tablet_bootstrap.cc:492] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902: Bootstrap starting.
I20240502 08:16:47.893118 9024 tablet_bootstrap.cc:654] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.894104 9024 log.cc:824] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:47.896411 9024 tablet_bootstrap.cc:492] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902: No bootstrap required, opened a new log
I20240502 08:16:47.896651 9024 ts_tablet_manager.cc:1367] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 08:16:47.902729 9024 raft_consensus.cc:348] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:16:47.903016 9024 raft_consensus.cc:374] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:47.903096 9024 raft_consensus.cc:729] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58bd494983a4a2885cda3387b1be902, State: Initialized, Role: FOLLOWER
I20240502 08:16:47.903616 9024 consensus_queue.cc:260] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:16:47.903795 9024 raft_consensus.cc:388] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:47.903892 9024 raft_consensus.cc:482] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:47.904026 9024 raft_consensus.cc:3037] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.905834 9024 raft_consensus.cc:504] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:16:47.906245 9024 leader_election.cc:302] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b58bd494983a4a2885cda3387b1be902; no voters:
I20240502 08:16:47.906764 9024 leader_election.cc:288] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:47.906880 9026 raft_consensus.cc:2781] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:47.907514 9026 raft_consensus.cc:686] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 1 LEADER]: Becoming Leader. State: Replica: b58bd494983a4a2885cda3387b1be902, State: Running, Role: LEADER
I20240502 08:16:47.908044 9026 consensus_queue.cc:237] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:16:47.908447 9024 ts_tablet_manager.cc:1398] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20240502 08:16:47.912894 8044 catalog_manager.cc:5484] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 reported cstate change: term changed from 0 to 1, leader changed from <none> to b58bd494983a4a2885cda3387b1be902 (127.6.5.4). New cstate: current_term: 1 leader_uuid: "b58bd494983a4a2885cda3387b1be902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } } }
I20240502 08:16:47.923543 6164 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 08:16:47.925225 6164 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b90db359bc744f2b82b69d97627fa7b9 to finish bootstrapping
I20240502 08:16:47.933764 6164 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 89ae15f3455446339957f9559f3857da to finish bootstrapping
I20240502 08:16:47.941393 6164 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 00b1227597554ea18f7e125de6d2760e to finish bootstrapping
I20240502 08:16:47.949045 6164 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b58bd494983a4a2885cda3387b1be902 to finish bootstrapping
I20240502 08:16:47.955687 6164 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c4e97cd51e9949339f937b6be4cefebb to finish bootstrapping
I20240502 08:16:47.963191 6164 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 646203131e6c42a2a2e631374efbe5ee to finish bootstrapping
I20240502 08:16:47.970711 6164 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6e2d5eaefade459c8e1fce0db41f2aca to finish bootstrapping
I20240502 08:16:47.978123 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 8363
I20240502 08:16:47.979029 8485 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:48.005262 8750 heartbeater.cc:503] Master 127.6.5.62:44177 was elected leader, sending a full tablet report...
I20240502 08:16:48.059587 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 8363
I20240502 08:16:48.073088 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 8493
I20240502 08:16:48.074149 8615 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:48.157099 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 8493
I20240502 08:16:48.170476 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 8623
I20240502 08:16:48.171605 8745 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:48.250258 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 8623
I20240502 08:16:48.262579 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 8753
I20240502 08:16:48.263598 8875 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:48.348049 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 8753
I20240502 08:16:48.360366 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 8883
I20240502 08:16:48.361366 9005 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:16:48.443900 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 8883
I20240502 08:16:52.207942 8044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42606:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:52.226899 8295 tablet_service.cc:1467] Processing CreateTablet for tablet d9a643e4d8dc4eeba0a31fc105c141a0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=bfa22e199f7d4cd1aa040b3473937ac8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.226917 8165 tablet_service.cc:1467] Processing CreateTablet for tablet dba45a53e0ea4de1b03d12e36a8bb7bb (DEFAULT_TABLE table=rebalance_test_table_0 [id=bfa22e199f7d4cd1aa040b3473937ac8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.226953 8164 tablet_service.cc:1467] Processing CreateTablet for tablet bdce98c845724dc3a9cedf89e3bec62c (DEFAULT_TABLE table=rebalance_test_table_0 [id=bfa22e199f7d4cd1aa040b3473937ac8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.228181 8295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a643e4d8dc4eeba0a31fc105c141a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.228404 8164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdce98c845724dc3a9cedf89e3bec62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.228883 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba45a53e0ea4de1b03d12e36a8bb7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.238416 9051 tablet_bootstrap.cc:492] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da: Bootstrap starting.
I20240502 08:16:52.244094 9052 tablet_bootstrap.cc:492] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9: Bootstrap starting.
I20240502 08:16:52.244545 9051 tablet_bootstrap.cc:654] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.245884 9051 log.cc:824] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:52.247310 9052 tablet_bootstrap.cc:654] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.248474 9052 log.cc:824] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:52.248719 9051 tablet_bootstrap.cc:492] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da: No bootstrap required, opened a new log
I20240502 08:16:52.249002 9051 ts_tablet_manager.cc:1367] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20240502 08:16:52.251024 9052 tablet_bootstrap.cc:492] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9: No bootstrap required, opened a new log
I20240502 08:16:52.251263 9052 ts_tablet_manager.cc:1367] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 08:16:52.257247 9051 raft_consensus.cc:348] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:52.257601 9051 raft_consensus.cc:374] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.257699 9051 raft_consensus.cc:729] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ae15f3455446339957f9559f3857da, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.258375 9051 consensus_queue.cc:260] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:52.258631 9051 raft_consensus.cc:388] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:52.258757 9051 raft_consensus.cc:482] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:52.258919 9051 raft_consensus.cc:3037] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.259452 9052 raft_consensus.cc:348] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:52.259783 9052 raft_consensus.cc:374] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.259881 9052 raft_consensus.cc:729] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.260468 9052 consensus_queue.cc:260] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:52.260694 9052 raft_consensus.cc:388] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:52.260816 9052 raft_consensus.cc:482] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:52.260954 9052 raft_consensus.cc:3037] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.261524 9051 raft_consensus.cc:504] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:52.262054 9051 leader_election.cc:302] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ae15f3455446339957f9559f3857da; no voters:
I20240502 08:16:52.262676 9051 leader_election.cc:288] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:52.262779 9055 raft_consensus.cc:2781] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.263587 9052 raft_consensus.cc:504] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:52.263880 9055 raft_consensus.cc:686] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Becoming Leader. State: Replica: 89ae15f3455446339957f9559f3857da, State: Running, Role: LEADER
I20240502 08:16:52.264139 9052 leader_election.cc:302] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90db359bc744f2b82b69d97627fa7b9; no voters:
I20240502 08:16:52.264390 9055 consensus_queue.cc:237] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:52.264832 9051 ts_tablet_manager.cc:1398] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20240502 08:16:52.265578 9052 leader_election.cc:288] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:52.265792 9058 raft_consensus.cc:2781] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.270673 9052 ts_tablet_manager.cc:1398] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9: Time spent starting tablet: real 0.019s user 0.016s sys 0.003s
I20240502 08:16:52.271503 9052 tablet_bootstrap.cc:492] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9: Bootstrap starting.
I20240502 08:16:52.273274 9052 tablet_bootstrap.cc:654] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.274905 9058 raft_consensus.cc:686] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Leader. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:16:52.274981 8044 catalog_manager.cc:5484] T d9a643e4d8dc4eeba0a31fc105c141a0 P 89ae15f3455446339957f9559f3857da reported cstate change: term changed from 0 to 1, leader changed from <none> to 89ae15f3455446339957f9559f3857da (127.6.5.2). New cstate: current_term: 1 leader_uuid: "89ae15f3455446339957f9559f3857da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } }
I20240502 08:16:52.275424 9058 consensus_queue.cc:237] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:52.277348 9052 tablet_bootstrap.cc:492] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9: No bootstrap required, opened a new log
I20240502 08:16:52.277472 9052 ts_tablet_manager.cc:1367] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240502 08:16:52.278122 9052 raft_consensus.cc:348] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:52.278240 9052 raft_consensus.cc:374] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.278328 9052 raft_consensus.cc:729] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.278591 9052 consensus_queue.cc:260] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:52.278784 9052 raft_consensus.cc:388] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:52.278878 9052 raft_consensus.cc:482] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:52.278985 9052 raft_consensus.cc:3037] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.280155 8044 catalog_manager.cc:5484] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1). New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } }
I20240502 08:16:52.281627 9052 raft_consensus.cc:504] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:52.281889 9052 leader_election.cc:302] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90db359bc744f2b82b69d97627fa7b9; no voters:
I20240502 08:16:52.282285 9052 leader_election.cc:288] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:52.282390 9061 raft_consensus.cc:2781] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.283105 9052 ts_tablet_manager.cc:1398] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:16:52.283533 9061 raft_consensus.cc:686] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Leader. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:16:52.283804 9061 consensus_queue.cc:237] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:52.285830 8043 catalog_manager.cc:5484] T dba45a53e0ea4de1b03d12e36a8bb7bb P b90db359bc744f2b82b69d97627fa7b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1). New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } }
W20240502 08:16:52.605985 9068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:52.606280 9068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:52.613607 9068 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 110.34 ms
time per row: 0.05517 ms
I20240502 08:16:52.859416 8042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42606:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:52.867265 8295 tablet_service.cc:1467] Processing CreateTablet for tablet c772b9342a314c3191a5caaf4395c94f (DEFAULT_TABLE table=rebalance_test_table_1 [id=37605e3d381045239bd078121ab54c76]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.867367 8294 tablet_service.cc:1467] Processing CreateTablet for tablet 1fabef91cc4d4df3abfe3bdb393790e8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=37605e3d381045239bd078121ab54c76]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.867710 8295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c772b9342a314c3191a5caaf4395c94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.867995 8294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fabef91cc4d4df3abfe3bdb393790e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.868412 8165 tablet_service.cc:1467] Processing CreateTablet for tablet f87d008812d847bb89dcf4034387bb51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=37605e3d381045239bd078121ab54c76]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:52.868806 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87d008812d847bb89dcf4034387bb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:52.873464 9106 tablet_bootstrap.cc:492] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da: Bootstrap starting.
I20240502 08:16:52.874662 9107 tablet_bootstrap.cc:492] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9: Bootstrap starting.
I20240502 08:16:52.874909 9106 tablet_bootstrap.cc:654] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.875885 9107 tablet_bootstrap.cc:654] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.876694 9106 tablet_bootstrap.cc:492] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da: No bootstrap required, opened a new log
I20240502 08:16:52.876827 9106 ts_tablet_manager.cc:1367] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:52.877411 9107 tablet_bootstrap.cc:492] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9: No bootstrap required, opened a new log
I20240502 08:16:52.877534 9107 ts_tablet_manager.cc:1367] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:52.877532 9106 raft_consensus.cc:348] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:52.877686 9106 raft_consensus.cc:374] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.877770 9106 raft_consensus.cc:729] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ae15f3455446339957f9559f3857da, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.877980 9106 consensus_queue.cc:260] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:52.878118 9106 raft_consensus.cc:388] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:52.878188 9106 raft_consensus.cc:482] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:52.878196 9107 raft_consensus.cc:348] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:52.878283 9106 raft_consensus.cc:3037] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.878350 9107 raft_consensus.cc:374] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.878436 9107 raft_consensus.cc:729] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.878665 9107 consensus_queue.cc:260] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:52.878806 9107 raft_consensus.cc:388] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:52.878880 9107 raft_consensus.cc:482] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:52.878975 9107 raft_consensus.cc:3037] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.880093 9106 raft_consensus.cc:504] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:52.880333 9106 leader_election.cc:302] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ae15f3455446339957f9559f3857da; no voters:
I20240502 08:16:52.880508 9106 leader_election.cc:288] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:52.880652 9055 raft_consensus.cc:2781] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.880960 9107 raft_consensus.cc:504] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:52.881098 9055 raft_consensus.cc:686] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Becoming Leader. State: Replica: 89ae15f3455446339957f9559f3857da, State: Running, Role: LEADER
I20240502 08:16:52.881197 9107 leader_election.cc:302] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90db359bc744f2b82b69d97627fa7b9; no voters:
I20240502 08:16:52.881422 9061 raft_consensus.cc:2781] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.881340 9055 consensus_queue.cc:237] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:52.881847 9061 raft_consensus.cc:686] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Leader. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:16:52.882068 9061 consensus_queue.cc:237] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:52.883178 9106 ts_tablet_manager.cc:1398] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20240502 08:16:52.883419 9106 tablet_bootstrap.cc:492] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da: Bootstrap starting.
I20240502 08:16:52.883649 8042 catalog_manager.cc:5484] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da reported cstate change: term changed from 0 to 1, leader changed from <none> to 89ae15f3455446339957f9559f3857da (127.6.5.2). New cstate: current_term: 1 leader_uuid: "89ae15f3455446339957f9559f3857da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } }
I20240502 08:16:52.884795 9106 tablet_bootstrap.cc:654] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:52.887066 9107 leader_election.cc:288] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:52.887498 9107 ts_tablet_manager.cc:1398] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9: Time spent starting tablet: real 0.010s user 0.001s sys 0.004s
I20240502 08:16:52.887432 8042 catalog_manager.cc:5484] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1). New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } }
I20240502 08:16:52.888283 9106 tablet_bootstrap.cc:492] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da: No bootstrap required, opened a new log
I20240502 08:16:52.888411 9106 ts_tablet_manager.cc:1367] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:16:52.888969 9106 raft_consensus.cc:348] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:52.889142 9106 raft_consensus.cc:374] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:52.889652 9106 raft_consensus.cc:729] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ae15f3455446339957f9559f3857da, State: Initialized, Role: FOLLOWER
I20240502 08:16:52.889915 9106 consensus_queue.cc:260] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:52.890092 9106 raft_consensus.cc:388] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:52.890165 9106 raft_consensus.cc:482] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:52.890245 9106 raft_consensus.cc:3037] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:52.892282 9106 raft_consensus.cc:504] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:52.892452 9106 leader_election.cc:302] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ae15f3455446339957f9559f3857da; no voters:
I20240502 08:16:52.892649 9106 leader_election.cc:288] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:52.892720 9110 raft_consensus.cc:2781] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:52.892865 9110 raft_consensus.cc:686] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Becoming Leader. State: Replica: 89ae15f3455446339957f9559f3857da, State: Running, Role: LEADER
I20240502 08:16:52.893031 9106 ts_tablet_manager.cc:1398] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:16:52.893151 9110 consensus_queue.cc:237] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:52.895385 8040 catalog_manager.cc:5484] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da reported cstate change: term changed from 0 to 1, leader changed from <none> to 89ae15f3455446339957f9559f3857da (127.6.5.2). New cstate: current_term: 1 leader_uuid: "89ae15f3455446339957f9559f3857da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } }
W20240502 08:16:53.198792 9117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:53.199087 9117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:53.206020 9117 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 117.277 ms
time per row: 0.0586387 ms
I20240502 08:16:53.451197 8044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42606:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:53.459066 8165 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3e2b0c939e4182bb851b4cdc802301 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ec0e8263e141487d845dc940a90656fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.459169 8295 tablet_service.cc:1467] Processing CreateTablet for tablet 4a905a7206774ff096194077781dcd45 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ec0e8263e141487d845dc940a90656fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.459529 8295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a905a7206774ff096194077781dcd45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.459532 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3e2b0c939e4182bb851b4cdc802301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.459463 8164 tablet_service.cc:1467] Processing CreateTablet for tablet 72b8cb6631624a839ec2b3a77abaa1df (DEFAULT_TABLE table=rebalance_test_table_2 [id=ec0e8263e141487d845dc940a90656fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.459825 8164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b8cb6631624a839ec2b3a77abaa1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.465497 9155 tablet_bootstrap.cc:492] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9: Bootstrap starting.
I20240502 08:16:53.465953 9156 tablet_bootstrap.cc:492] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da: Bootstrap starting.
I20240502 08:16:53.466842 9155 tablet_bootstrap.cc:654] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.467145 9156 tablet_bootstrap.cc:654] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.468446 9155 tablet_bootstrap.cc:492] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9: No bootstrap required, opened a new log
I20240502 08:16:53.468633 9155 ts_tablet_manager.cc:1367] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:53.468755 9156 tablet_bootstrap.cc:492] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da: No bootstrap required, opened a new log
I20240502 08:16:53.468878 9156 ts_tablet_manager.cc:1367] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:53.469334 9155 raft_consensus.cc:348] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:53.469521 9155 raft_consensus.cc:374] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.469547 9156 raft_consensus.cc:348] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:53.469619 9155 raft_consensus.cc:729] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.469714 9156 raft_consensus.cc:374] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.469801 9156 raft_consensus.cc:729] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ae15f3455446339957f9559f3857da, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.469843 9155 consensus_queue.cc:260] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:53.470026 9155 raft_consensus.cc:388] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:53.470018 9156 consensus_queue.cc:260] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:53.470100 9155 raft_consensus.cc:482] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:53.470196 9155 raft_consensus.cc:3037] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.470196 9156 raft_consensus.cc:388] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:53.470271 9156 raft_consensus.cc:482] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:53.470374 9156 raft_consensus.cc:3037] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.472643 9155 raft_consensus.cc:504] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:53.472795 9156 raft_consensus.cc:504] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:53.472889 9155 leader_election.cc:302] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90db359bc744f2b82b69d97627fa7b9; no voters:
I20240502 08:16:53.473018 9156 leader_election.cc:302] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ae15f3455446339957f9559f3857da; no voters:
I20240502 08:16:53.473065 9155 leader_election.cc:288] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:53.473156 9061 raft_consensus.cc:2781] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:53.473237 9156 leader_election.cc:288] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:53.473299 9111 raft_consensus.cc:2781] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:53.473670 9156 ts_tablet_manager.cc:1398] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:16:53.474701 9111 raft_consensus.cc:686] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Becoming Leader. State: Replica: 89ae15f3455446339957f9559f3857da, State: Running, Role: LEADER
I20240502 08:16:53.474947 9061 raft_consensus.cc:686] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Leader. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:16:53.474979 9111 consensus_queue.cc:237] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:53.475198 9061 consensus_queue.cc:237] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:53.476778 9155 ts_tablet_manager.cc:1398] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20240502 08:16:53.477020 9155 tablet_bootstrap.cc:492] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9: Bootstrap starting.
I20240502 08:16:53.477268 8044 catalog_manager.cc:5484] T 4a905a7206774ff096194077781dcd45 P 89ae15f3455446339957f9559f3857da reported cstate change: term changed from 0 to 1, leader changed from <none> to 89ae15f3455446339957f9559f3857da (127.6.5.2). New cstate: current_term: 1 leader_uuid: "89ae15f3455446339957f9559f3857da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } }
I20240502 08:16:53.478416 9155 tablet_bootstrap.cc:654] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.480216 8039 catalog_manager.cc:5484] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1). New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } }
I20240502 08:16:53.482409 9155 tablet_bootstrap.cc:492] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9: No bootstrap required, opened a new log
I20240502 08:16:53.482571 9155 ts_tablet_manager.cc:1367] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240502 08:16:53.483135 9155 raft_consensus.cc:348] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:53.483275 9155 raft_consensus.cc:374] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.483350 9155 raft_consensus.cc:729] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.483561 9155 consensus_queue.cc:260] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:53.483685 9155 raft_consensus.cc:388] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:53.483764 9155 raft_consensus.cc:482] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:53.483841 9155 raft_consensus.cc:3037] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.485807 9155 raft_consensus.cc:504] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:53.485992 9155 leader_election.cc:302] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90db359bc744f2b82b69d97627fa7b9; no voters:
I20240502 08:16:53.486197 9155 leader_election.cc:288] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:53.486274 9159 raft_consensus.cc:2781] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:53.486429 9159 raft_consensus.cc:686] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Leader. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:16:53.486626 9155 ts_tablet_manager.cc:1398] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240502 08:16:53.486677 9159 consensus_queue.cc:237] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:53.488497 8039 catalog_manager.cc:5484] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1). New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } }
W20240502 08:16:53.788666 9166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:53.788954 9166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:53.796068 9166 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 111.011 ms
time per row: 0.0555057 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:16:54.888670 8043 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42606:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:54.896963 8295 tablet_service.cc:1467] Processing CreateTablet for tablet aa41e5f5e3ef4dec89a0a39198f7a53c (DEFAULT_TABLE table=rebalance_test_table_3 [id=d999095ea3574127ad81d93a2c85019d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.897118 8294 tablet_service.cc:1467] Processing CreateTablet for tablet 3d96e8d4b891410980ae6a696799a60b (DEFAULT_TABLE table=rebalance_test_table_3 [id=d999095ea3574127ad81d93a2c85019d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.897333 8164 tablet_service.cc:1467] Processing CreateTablet for tablet 9802b878c2f6463983a4029b7031004d (DEFAULT_TABLE table=rebalance_test_table_3 [id=d999095ea3574127ad81d93a2c85019d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.897449 8295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa41e5f5e3ef4dec89a0a39198f7a53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.897711 8164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9802b878c2f6463983a4029b7031004d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.897722 8294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d96e8d4b891410980ae6a696799a60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.903676 9209 tablet_bootstrap.cc:492] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9: Bootstrap starting.
I20240502 08:16:54.904042 9210 tablet_bootstrap.cc:492] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da: Bootstrap starting.
I20240502 08:16:54.905004 9209 tablet_bootstrap.cc:654] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.905164 9210 tablet_bootstrap.cc:654] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.906647 9210 tablet_bootstrap.cc:492] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da: No bootstrap required, opened a new log
I20240502 08:16:54.906831 9210 ts_tablet_manager.cc:1367] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:54.906961 9209 tablet_bootstrap.cc:492] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9: No bootstrap required, opened a new log
I20240502 08:16:54.907089 9209 ts_tablet_manager.cc:1367] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:54.907450 9210 raft_consensus.cc:348] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:54.907580 9210 raft_consensus.cc:374] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.907680 9210 raft_consensus.cc:729] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ae15f3455446339957f9559f3857da, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.907675 9209 raft_consensus.cc:348] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:54.907830 9209 raft_consensus.cc:374] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.907919 9209 raft_consensus.cc:729] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.907888 9210 consensus_queue.cc:260] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:54.908051 9210 raft_consensus.cc:388] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:54.908120 9210 raft_consensus.cc:482] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:54.908126 9209 consensus_queue.cc:260] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:54.908215 9210 raft_consensus.cc:3037] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.908309 9209 raft_consensus.cc:388] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:54.908380 9209 raft_consensus.cc:482] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:54.908480 9209 raft_consensus.cc:3037] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.909896 9210 raft_consensus.cc:504] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:54.910123 9210 leader_election.cc:302] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ae15f3455446339957f9559f3857da; no voters:
I20240502 08:16:54.910142 9209 raft_consensus.cc:504] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:54.910337 9209 leader_election.cc:302] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90db359bc744f2b82b69d97627fa7b9; no voters:
I20240502 08:16:54.910833 9209 leader_election.cc:288] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:54.910995 9214 raft_consensus.cc:2781] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.911626 9209 ts_tablet_manager.cc:1398] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 08:16:54.911743 9214 raft_consensus.cc:686] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Leader. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:16:54.912000 9214 consensus_queue.cc:237] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:54.914204 8043 catalog_manager.cc:5484] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1). New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } }
I20240502 08:16:54.914685 9210 leader_election.cc:288] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:54.914803 9213 raft_consensus.cc:2781] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.915416 9210 ts_tablet_manager.cc:1398] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20240502 08:16:54.915745 9210 tablet_bootstrap.cc:492] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da: Bootstrap starting.
I20240502 08:16:54.915714 9213 raft_consensus.cc:686] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Becoming Leader. State: Replica: 89ae15f3455446339957f9559f3857da, State: Running, Role: LEADER
I20240502 08:16:54.915990 9213 consensus_queue.cc:237] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:54.917238 9210 tablet_bootstrap.cc:654] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.919137 9210 tablet_bootstrap.cc:492] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da: No bootstrap required, opened a new log
I20240502 08:16:54.919273 9210 ts_tablet_manager.cc:1367] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 08:16:54.919826 9210 raft_consensus.cc:348] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:54.919971 9210 raft_consensus.cc:374] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.919857 8043 catalog_manager.cc:5484] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da reported cstate change: term changed from 0 to 1, leader changed from <none> to 89ae15f3455446339957f9559f3857da (127.6.5.2). New cstate: current_term: 1 leader_uuid: "89ae15f3455446339957f9559f3857da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } }
I20240502 08:16:54.920042 9210 raft_consensus.cc:729] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ae15f3455446339957f9559f3857da, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.920229 9210 consensus_queue.cc:260] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:54.920354 9210 raft_consensus.cc:388] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:54.920436 9210 raft_consensus.cc:482] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:54.920517 9210 raft_consensus.cc:3037] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.922488 9210 raft_consensus.cc:504] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:54.922740 9210 leader_election.cc:302] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ae15f3455446339957f9559f3857da; no voters:
I20240502 08:16:54.922940 9210 leader_election.cc:288] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:54.923101 9213 raft_consensus.cc:2781] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.923275 9213 raft_consensus.cc:686] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Becoming Leader. State: Replica: 89ae15f3455446339957f9559f3857da, State: Running, Role: LEADER
I20240502 08:16:54.923386 9210 ts_tablet_manager.cc:1398] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20240502 08:16:54.923476 9213 consensus_queue.cc:237] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:54.925770 8043 catalog_manager.cc:5484] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da reported cstate change: term changed from 0 to 1, leader changed from <none> to 89ae15f3455446339957f9559f3857da (127.6.5.2). New cstate: current_term: 1 leader_uuid: "89ae15f3455446339957f9559f3857da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } }
W20240502 08:16:55.222502 9225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:55.222776 9225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:55.229718 9225 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 104.206 ms
time per row: 0.0521031 ms
I20240502 08:16:55.463814 8043 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42606:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:16:55.471606 8164 tablet_service.cc:1467] Processing CreateTablet for tablet afc5bc8efa7d4a3fb8970bf100c754a2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=bf5ea02dca1c4ab89d3224c8e4a45df1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.471657 8295 tablet_service.cc:1467] Processing CreateTablet for tablet 74a88bd56f86453280555608afcb98f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=bf5ea02dca1c4ab89d3224c8e4a45df1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.472072 8164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc5bc8efa7d4a3fb8970bf100c754a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.472074 8295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a88bd56f86453280555608afcb98f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.472452 8165 tablet_service.cc:1467] Processing CreateTablet for tablet f766891d6cea4e1e827d3233c2e17a27 (DEFAULT_TABLE table=rebalance_test_table_4 [id=bf5ea02dca1c4ab89d3224c8e4a45df1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:55.472813 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f766891d6cea4e1e827d3233c2e17a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:55.477958 9265 tablet_bootstrap.cc:492] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da: Bootstrap starting.
I20240502 08:16:55.478119 9264 tablet_bootstrap.cc:492] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9: Bootstrap starting.
I20240502 08:16:55.479228 9265 tablet_bootstrap.cc:654] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.479549 9264 tablet_bootstrap.cc:654] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.480832 9265 tablet_bootstrap.cc:492] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da: No bootstrap required, opened a new log
I20240502 08:16:55.480957 9265 ts_tablet_manager.cc:1367] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:16:55.481491 9265 raft_consensus.cc:348] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:55.481624 9265 raft_consensus.cc:374] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.481709 9265 raft_consensus.cc:729] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ae15f3455446339957f9559f3857da, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.481896 9265 consensus_queue.cc:260] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:55.482054 9265 raft_consensus.cc:388] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:55.482118 9265 raft_consensus.cc:482] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:55.482216 9265 raft_consensus.cc:3037] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.482604 9264 tablet_bootstrap.cc:492] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9: No bootstrap required, opened a new log
I20240502 08:16:55.482717 9264 ts_tablet_manager.cc:1367] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:16:55.483229 9264 raft_consensus.cc:348] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:55.483359 9264 raft_consensus.cc:374] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.483484 9264 raft_consensus.cc:729] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.483686 9264 consensus_queue.cc:260] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:55.483834 9264 raft_consensus.cc:388] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:55.483906 9264 raft_consensus.cc:482] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:55.484006 9264 raft_consensus.cc:3037] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.484387 9265 raft_consensus.cc:504] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:55.484592 9265 leader_election.cc:302] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ae15f3455446339957f9559f3857da; no voters:
I20240502 08:16:55.484746 9265 leader_election.cc:288] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:55.484849 9220 raft_consensus.cc:2781] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:55.485162 9265 ts_tablet_manager.cc:1398] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240502 08:16:55.485132 9220 raft_consensus.cc:686] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Becoming Leader. State: Replica: 89ae15f3455446339957f9559f3857da, State: Running, Role: LEADER
I20240502 08:16:55.485414 9220 consensus_queue.cc:237] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:16:55.486100 9264 raft_consensus.cc:504] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:55.486310 9264 leader_election.cc:302] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90db359bc744f2b82b69d97627fa7b9; no voters:
I20240502 08:16:55.486497 9264 leader_election.cc:288] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:55.486547 9214 raft_consensus.cc:2781] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:55.487160 9264 ts_tablet_manager.cc:1398] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20240502 08:16:55.487514 9214 raft_consensus.cc:686] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Leader. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:16:55.487818 8043 catalog_manager.cc:5484] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da reported cstate change: term changed from 0 to 1, leader changed from <none> to 89ae15f3455446339957f9559f3857da (127.6.5.2). New cstate: current_term: 1 leader_uuid: "89ae15f3455446339957f9559f3857da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } }
I20240502 08:16:55.488044 9264 tablet_bootstrap.cc:492] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9: Bootstrap starting.
I20240502 08:16:55.487967 9214 consensus_queue.cc:237] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:55.489663 9264 tablet_bootstrap.cc:654] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:55.491544 8038 catalog_manager.cc:5484] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1). New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } }
I20240502 08:16:55.493041 9264 tablet_bootstrap.cc:492] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9: No bootstrap required, opened a new log
I20240502 08:16:55.493332 9264 ts_tablet_manager.cc:1367] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s
I20240502 08:16:55.493878 9264 raft_consensus.cc:348] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:55.494009 9264 raft_consensus.cc:374] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:55.494073 9264 raft_consensus.cc:729] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Initialized, Role: FOLLOWER
I20240502 08:16:55.494267 9264 consensus_queue.cc:260] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:55.494386 9264 raft_consensus.cc:388] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:55.494498 9264 raft_consensus.cc:482] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:55.494576 9264 raft_consensus.cc:3037] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:55.496548 9264 raft_consensus.cc:504] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:55.496731 9264 leader_election.cc:302] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90db359bc744f2b82b69d97627fa7b9; no voters:
I20240502 08:16:55.496903 9264 leader_election.cc:288] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:16:55.496958 9270 raft_consensus.cc:2781] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:55.497160 9270 raft_consensus.cc:686] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Leader. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:16:55.497283 9264 ts_tablet_manager.cc:1398] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:16:55.497449 9270 consensus_queue.cc:237] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:55.499217 8038 catalog_manager.cc:5484] T f766891d6cea4e1e827d3233c2e17a27 P b90db359bc744f2b82b69d97627fa7b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1). New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } }
W20240502 08:16:55.800451 9274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:55.800736 9274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:55.807730 9274 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 92.2777 ms
time per row: 0.0461389 ms
I20240502 08:16:56.025238 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.3:43567
--local_ip_for_outbound_sockets=127.6.5.3
--tserver_master_addrs=127.6.5.62:44177
--webserver_port=33467
--webserver_interface=127.6.5.3
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:16:56.332381 9311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:56.332645 9311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:56.332778 9311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:56.339818 9311 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:56.339919 9311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:56.340075 9311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.3
I20240502 08:16:56.348482 9311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.3:43567
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.5.3
--webserver_port=33467
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.3
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:56.349578 9311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:56.351183 9311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:56.362581 9316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:56.363034 9319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:56.363428 9317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:56.363919 9311 server_base.cc:1034] running on GCE node
I20240502 08:16:56.364435 9311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:56.365257 9311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:56.366477 9311 hybrid_clock.cc:648] HybridClock initialized: now 1714637816366419 us; error 38 us; skew 500 ppm
I20240502 08:16:56.366772 9311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:56.368829 9311 webserver.cc:460] Webserver started at http://127.6.5.3:33467/ using document root <none> and password file <none>
I20240502 08:16:56.369537 9311 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:56.369655 9311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:56.374357 9311 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:16:56.376927 9325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:56.378082 9311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:16:56.378208 9311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "00b1227597554ea18f7e125de6d2760e"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-h7h6"
I20240502 08:16:56.378940 9311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:56.395982 9311 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:56.396615 9311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:56.396854 9311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:56.397670 9311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:56.399011 9311 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:56.399078 9311 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:56.399164 9311 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:56.399205 9311 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:56.425583 9311 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.3:43567
I20240502 08:16:56.425609 9437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.3:43567 every 8 connection(s)
I20240502 08:16:56.427037 9311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:16:56.428218 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 9311
I20240502 08:16:56.428835 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.4:35529
--local_ip_for_outbound_sockets=127.6.5.4
--tserver_master_addrs=127.6.5.62:44177
--webserver_port=35027
--webserver_interface=127.6.5.4
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:56.440718 9438 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:16:56.441129 9438 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:56.441855 9438 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:16:56.443327 8039 ts_manager.cc:194] Re-registered known tserver with Master: 00b1227597554ea18f7e125de6d2760e (127.6.5.3:43567)
I20240502 08:16:56.444190 8039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.3:35679
W20240502 08:16:56.734202 9442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:56.734486 9442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:56.734611 9442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:56.741670 9442 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:56.741770 9442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:56.741946 9442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.4
I20240502 08:16:56.750830 9442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.4:35529
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.5.4
--webserver_port=35027
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.4
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:56.751941 9442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:56.753509 9442 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:56.765039 9447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:56.765621 9448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:56.765944 9450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:56.766438 9442 server_base.cc:1034] running on GCE node
I20240502 08:16:56.766999 9442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:56.767833 9442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:56.769016 9442 hybrid_clock.cc:648] HybridClock initialized: now 1714637816768971 us; error 43 us; skew 500 ppm
I20240502 08:16:56.769336 9442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:56.771939 9442 webserver.cc:460] Webserver started at http://127.6.5.4:35027/ using document root <none> and password file <none>
I20240502 08:16:56.772599 9442 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:56.772717 9442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:56.777434 9442 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:16:56.780010 9456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:56.781183 9442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:56.781335 9442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b58bd494983a4a2885cda3387b1be902"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-h7h6"
I20240502 08:16:56.782159 9442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:56.815083 9442 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:56.815812 9442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:56.816068 9442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:56.816905 9442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:56.818665 9463 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 08:16:56.823410 9442 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 08:16:56.823496 9442 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20240502 08:16:56.823626 9442 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 08:16:56.826217 9442 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 08:16:56.826309 9442 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20240502 08:16:56.826653 9463 tablet_bootstrap.cc:492] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902: Bootstrap starting.
I20240502 08:16:56.845072 9463 log.cc:824] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:56.849167 9463 tablet_bootstrap.cc:492] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:16:56.849948 9463 tablet_bootstrap.cc:492] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902: Bootstrap complete.
I20240502 08:16:56.851286 9463 ts_tablet_manager.cc:1367] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.014s
I20240502 08:16:56.861130 9463 raft_consensus.cc:348] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:16:56.862025 9463 raft_consensus.cc:729] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58bd494983a4a2885cda3387b1be902, State: Initialized, Role: FOLLOWER
I20240502 08:16:56.862238 9442 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.4:35529
I20240502 08:16:56.862285 9570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.4:35529 every 8 connection(s)
I20240502 08:16:56.862807 9463 consensus_queue.cc:260] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:16:56.863003 9463 raft_consensus.cc:388] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:56.863123 9463 raft_consensus.cc:482] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:56.863284 9463 raft_consensus.cc:3037] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:16:56.863566 9442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:16:56.865126 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 9442
I20240502 08:16:56.865715 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.5:34875
--local_ip_for_outbound_sockets=127.6.5.5
--tserver_master_addrs=127.6.5.62:44177
--webserver_port=42903
--webserver_interface=127.6.5.5
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:56.876950 9571 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:16:56.877193 9571 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:56.877853 9571 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:16:56.891014 9463 raft_consensus.cc:504] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:16:56.891513 9463 leader_election.cc:302] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b58bd494983a4a2885cda3387b1be902; no voters:
I20240502 08:16:56.894601 8039 ts_manager.cc:194] Re-registered known tserver with Master: b58bd494983a4a2885cda3387b1be902 (127.6.5.4:35529)
I20240502 08:16:56.895064 9463 leader_election.cc:288] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240502 08:16:56.895210 9575 raft_consensus.cc:2781] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:16:56.896217 8039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.4:40887
I20240502 08:16:56.899679 9575 raft_consensus.cc:686] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [term 2 LEADER]: Becoming Leader. State: Replica: b58bd494983a4a2885cda3387b1be902, State: Running, Role: LEADER
I20240502 08:16:56.900219 9571 heartbeater.cc:503] Master 127.6.5.62:44177 was elected leader, sending a full tablet report...
I20240502 08:16:56.900223 9575 consensus_queue.cc:237] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:16:56.901252 9463 ts_tablet_manager.cc:1398] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902: Time spent starting tablet: real 0.050s user 0.015s sys 0.011s
I20240502 08:16:56.903386 8039 catalog_manager.cc:5484] T da122388bec1424bb9e07919dd2d3cf0 P b58bd494983a4a2885cda3387b1be902 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "b58bd494983a4a2885cda3387b1be902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } } }
W20240502 08:16:57.178452 9574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:57.178735 9574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:57.178866 9574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:57.185961 9574 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:57.186056 9574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:57.186167 9574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.5
I20240502 08:16:57.194532 9574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.5:34875
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.5.5
--webserver_port=42903
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.5
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:57.195659 9574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:57.197268 9574 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:57.209733 9588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:57.210183 9589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:57.210675 9591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:57.211179 9574 server_base.cc:1034] running on GCE node
I20240502 08:16:57.211789 9574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:57.212657 9574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:57.213850 9574 hybrid_clock.cc:648] HybridClock initialized: now 1714637817213814 us; error 26 us; skew 500 ppm
I20240502 08:16:57.214152 9574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:57.216127 9574 webserver.cc:460] Webserver started at http://127.6.5.5:42903/ using document root <none> and password file <none>
I20240502 08:16:57.216782 9574 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:57.216902 9574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:57.221720 9574 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:57.224222 9597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:57.225371 9574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:16:57.225509 9574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c4e97cd51e9949339f937b6be4cefebb"
format_stamp: "Formatted at 2024-05-02 08:16:46 on dist-test-slave-h7h6"
I20240502 08:16:57.226246 9574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:57.271147 9574 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:57.271921 9574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:57.272190 9574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:57.273048 9574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:57.274528 9574 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:57.274603 9574 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:57.274719 9574 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:57.274770 9574 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:57.302918 9574 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.5:34875
I20240502 08:16:57.302981 9709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.5:34875 every 8 connection(s)
I20240502 08:16:57.304229 9574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:16:57.304962 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 9574
I20240502 08:16:57.305495 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.6:34035
--local_ip_for_outbound_sockets=127.6.5.6
--tserver_master_addrs=127.6.5.62:44177
--webserver_port=39643
--webserver_interface=127.6.5.6
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:57.317241 9710 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:16:57.317530 9710 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:57.318187 9710 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:16:57.319588 8039 ts_manager.cc:194] Re-registered known tserver with Master: c4e97cd51e9949339f937b6be4cefebb (127.6.5.5:34875)
I20240502 08:16:57.320698 8039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.5:56971
I20240502 08:16:57.446974 9438 heartbeater.cc:503] Master 127.6.5.62:44177 was elected leader, sending a full tablet report...
W20240502 08:16:57.610193 9713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:57.610436 9713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:57.610575 9713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:57.617627 9713 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:57.617729 9713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:57.617870 9713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.6
I20240502 08:16:57.626166 9713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.6:34035
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.5.6
--webserver_port=39643
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.6
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:57.627235 9713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:57.628840 9713 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:57.640175 9719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:57.640717 9718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:57.641124 9721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:57.641667 9713 server_base.cc:1034] running on GCE node
I20240502 08:16:57.642195 9713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:57.643036 9713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:57.644230 9713 hybrid_clock.cc:648] HybridClock initialized: now 1714637817644185 us; error 37 us; skew 500 ppm
I20240502 08:16:57.644531 9713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:57.651086 9713 webserver.cc:460] Webserver started at http://127.6.5.6:39643/ using document root <none> and password file <none>
I20240502 08:16:57.651722 9713 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:57.651847 9713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:57.656653 9713 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 08:16:57.659117 9727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:57.660269 9713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:16:57.660409 9713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "646203131e6c42a2a2e631374efbe5ee"
format_stamp: "Formatted at 2024-05-02 08:16:47 on dist-test-slave-h7h6"
I20240502 08:16:57.661222 9713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:57.689226 9713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:57.689916 9713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:57.690174 9713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:57.691027 9713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:57.692386 9713 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:57.692462 9713 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:57.692582 9713 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:57.692632 9713 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:57.731387 9713 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.6:34035
I20240502 08:16:57.731478 9839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.6:34035 every 8 connection(s)
I20240502 08:16:57.732970 9713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:16:57.742142 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 9713
I20240502 08:16:57.742939 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.7:39789
--local_ip_for_outbound_sockets=127.6.5.7
--tserver_master_addrs=127.6.5.62:44177
--webserver_port=36201
--webserver_interface=127.6.5.7
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:16:57.748446 9840 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:16:57.748767 9840 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:57.749658 9840 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:16:57.751067 8039 ts_manager.cc:194] Re-registered known tserver with Master: 646203131e6c42a2a2e631374efbe5ee (127.6.5.6:34035)
I20240502 08:16:57.751834 8039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.6:45357
W20240502 08:16:58.051699 9843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:58.051949 9843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:58.052073 9843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:58.059161 9843 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:16:58.059275 9843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:58.059413 9843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.7
I20240502 08:16:58.067962 9843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.7:39789
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.5.7
--webserver_port=36201
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.7
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:58.069115 9843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:58.070772 9843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:58.082038 9849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:58.082547 9851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:58.083050 9848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:58.083595 9843 server_base.cc:1034] running on GCE node
I20240502 08:16:58.084136 9843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:58.084929 9843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:58.086107 9843 hybrid_clock.cc:648] HybridClock initialized: now 1714637818086057 us; error 39 us; skew 500 ppm
I20240502 08:16:58.086414 9843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:58.088454 9843 webserver.cc:460] Webserver started at http://127.6.5.7:36201/ using document root <none> and password file <none>
I20240502 08:16:58.089087 9843 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:58.089188 9843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:58.093909 9843 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:58.096415 9857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:58.097656 9843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:16:58.097797 9843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6e2d5eaefade459c8e1fce0db41f2aca"
format_stamp: "Formatted at 2024-05-02 08:16:47 on dist-test-slave-h7h6"
I20240502 08:16:58.098567 9843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:58.112454 9843 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:16:58.113128 9843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:58.113377 9843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:58.114185 9843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:58.115574 9843 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:58.115651 9843 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:58.115758 9843 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:58.115806 9843 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:16:58.143998 9843 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.7:39789
I20240502 08:16:58.144047 9969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.7:39789 every 8 connection(s)
I20240502 08:16:58.145500 9843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:16:58.150918 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 9843
I20240502 08:16:58.157914 9970 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:16:58.158226 9970 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:58.159021 9970 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:16:58.160373 8039 ts_manager.cc:194] Re-registered known tserver with Master: 6e2d5eaefade459c8e1fce0db41f2aca (127.6.5.7:39789)
I20240502 08:16:58.161301 8039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.7:35213
I20240502 08:16:58.323060 9710 heartbeater.cc:503] Master 127.6.5.62:44177 was elected leader, sending a full tablet report...
I20240502 08:16:58.603526 9905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.605204 9775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.607775 9373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.612100 8295 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:58.646595 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:58.655148 8165 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:16:58.689379 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.753952 9840 heartbeater.cc:503] Master 127.6.5.62:44177 was elected leader, sending a full tablet report...
I20240502 08:16:58.840924 9373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.846881 9905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.848698 9775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.864571 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:16:58.870632 8295 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:58.873796 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:16:58.883618 8165 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:16:58.979079 8185 consensus_queue.cc:237] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } }
W20240502 08:16:58.988123 8119 consensus_peers.cc:485] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 -> Peer c4e97cd51e9949339f937b6be4cefebb (127.6.5.5:34875): Couldn't send request to peer c4e97cd51e9949339f937b6be4cefebb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72b8cb6631624a839ec2b3a77abaa1df. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:16:59.164139 9970 heartbeater.cc:503] Master 127.6.5.62:44177 was elected leader, sending a full tablet report...
I20240502 08:16:59.476572 10072 ts_tablet_manager.cc:902] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb: Initiating tablet copy from peer b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:16:59.478543 10072 tablet_copy_client.cc:308] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:35591
I20240502 08:16:59.485734 8205 tablet_copy_service.cc:140] P b90db359bc744f2b82b69d97627fa7b9: Received BeginTabletCopySession request for tablet 72b8cb6631624a839ec2b3a77abaa1df from peer c4e97cd51e9949339f937b6be4cefebb ({username='slave'} at 127.6.5.5:36131)
I20240502 08:16:59.485941 8205 tablet_copy_service.cc:161] P b90db359bc744f2b82b69d97627fa7b9: Beginning new tablet copy session on tablet 72b8cb6631624a839ec2b3a77abaa1df from peer c4e97cd51e9949339f937b6be4cefebb at {username='slave'} at 127.6.5.5:36131: session id = c4e97cd51e9949339f937b6be4cefebb-72b8cb6631624a839ec2b3a77abaa1df
I20240502 08:16:59.488516 8205 tablet_copy_source_session.cc:205] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:16:59.492727 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b8cb6631624a839ec2b3a77abaa1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:59.501924 10072 tablet_copy_client.cc:783] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb: tablet copy: Starting download of 0 data blocks...
I20240502 08:16:59.502190 10072 tablet_copy_client.cc:647] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb: tablet copy: Starting download of 1 WAL segments...
I20240502 08:16:59.504474 10072 tablet_copy_client.cc:523] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:16:59.507623 10072 tablet_bootstrap.cc:492] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb: Bootstrap starting.
I20240502 08:16:59.516924 10072 log.cc:824] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:59.548941 10072 tablet_bootstrap.cc:492] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:16:59.549345 10072 tablet_bootstrap.cc:492] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb: Bootstrap complete.
I20240502 08:16:59.549685 10072 ts_tablet_manager.cc:1367] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb: Time spent bootstrapping tablet: real 0.042s user 0.039s sys 0.001s
I20240502 08:16:59.553720 10072 raft_consensus.cc:348] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:16:59.554473 10072 raft_consensus.cc:729] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e97cd51e9949339f937b6be4cefebb, State: Initialized, Role: FOLLOWER
I20240502 08:16:59.554865 10072 consensus_queue.cc:260] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } }
I20240502 08:16:59.556313 10072 ts_tablet_manager.cc:1398] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240502 08:16:59.557210 8205 tablet_copy_service.cc:342] P b90db359bc744f2b82b69d97627fa7b9: Request end of tablet copy session c4e97cd51e9949339f937b6be4cefebb-72b8cb6631624a839ec2b3a77abaa1df received from {username='slave'} at 127.6.5.5:36131
I20240502 08:16:59.557410 8205 tablet_copy_service.cc:433] P b90db359bc744f2b82b69d97627fa7b9: ending tablet copy session c4e97cd51e9949339f937b6be4cefebb-72b8cb6631624a839ec2b3a77abaa1df on tablet 72b8cb6631624a839ec2b3a77abaa1df with peer c4e97cd51e9949339f937b6be4cefebb
I20240502 08:16:59.983640 9665 raft_consensus.cc:1203] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:16:59.985388 10082 raft_consensus.cc:2932] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c4e97cd51e9949339f937b6be4cefebb (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } } }
I20240502 08:16:59.986064 9665 raft_consensus.cc:2932] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c4e97cd51e9949339f937b6be4cefebb (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } } }
I20240502 08:16:59.988572 10082 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46136 (request call id 0) took 1010 ms. Trace:
I20240502 08:16:59.988729 10082 rpcz_store.cc:276] 0502 08:16:58.978043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:16:58.978106 (+ 63us) service_pool.cc:225] Handling call
0502 08:16:58.980122 (+ 2016us) log.cc:842] Serialized 248 byte log entry
0502 08:16:59.988526 (+1008404us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":69,"raft.queue_time_us":347,"raft.run_cpu_time_us":483,"raft.run_wall_time_us":483,"spinlock_wait_cycles":83712,"thread_start_us":460,"threads_started":2,"wal-append.queue_time_us":316,"wal-append.run_cpu_time_us":578,"wal-append.run_wall_time_us":1000634}
I20240502 08:16:59.990990 8041 catalog_manager.cc:5484] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb reported cstate change: config changed from index -1 to 4, VOTER c4e97cd51e9949339f937b6be4cefebb (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } } }
I20240502 08:17:00.018232 8315 consensus_queue.cc:237] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } }
W20240502 08:17:00.027020 8249 consensus_peers.cc:485] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da -> Peer 6e2d5eaefade459c8e1fce0db41f2aca (127.6.5.7:39789): Couldn't send request to peer 6e2d5eaefade459c8e1fce0db41f2aca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa41e5f5e3ef4dec89a0a39198f7a53c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:00.407289 10110 ts_tablet_manager.cc:902] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca: Initiating tablet copy from peer 89ae15f3455446339957f9559f3857da (127.6.5.2:44539)
I20240502 08:17:00.408896 10110 tablet_copy_client.cc:308] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:44539
I20240502 08:17:00.415505 8335 tablet_copy_service.cc:140] P 89ae15f3455446339957f9559f3857da: Received BeginTabletCopySession request for tablet aa41e5f5e3ef4dec89a0a39198f7a53c from peer 6e2d5eaefade459c8e1fce0db41f2aca ({username='slave'} at 127.6.5.7:47405)
I20240502 08:17:00.415714 8335 tablet_copy_service.cc:161] P 89ae15f3455446339957f9559f3857da: Beginning new tablet copy session on tablet aa41e5f5e3ef4dec89a0a39198f7a53c from peer 6e2d5eaefade459c8e1fce0db41f2aca at {username='slave'} at 127.6.5.7:47405: session id = 6e2d5eaefade459c8e1fce0db41f2aca-aa41e5f5e3ef4dec89a0a39198f7a53c
I20240502 08:17:00.418162 8335 tablet_copy_source_session.cc:205] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:00.422428 10110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa41e5f5e3ef4dec89a0a39198f7a53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:00.432654 10110 tablet_copy_client.cc:783] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:00.432955 10110 tablet_copy_client.cc:647] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:00.435001 10110 tablet_copy_client.cc:523] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:00.437633 10110 tablet_bootstrap.cc:492] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca: Bootstrap starting.
I20240502 08:17:00.446333 10110 log.cc:824] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:00.475427 10110 tablet_bootstrap.cc:492] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:00.475832 10110 tablet_bootstrap.cc:492] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca: Bootstrap complete.
I20240502 08:17:00.476161 10110 ts_tablet_manager.cc:1367] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca: Time spent bootstrapping tablet: real 0.039s user 0.039s sys 0.000s
I20240502 08:17:00.480309 10110 raft_consensus.cc:348] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:17:00.481061 10110 raft_consensus.cc:729] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2d5eaefade459c8e1fce0db41f2aca, State: Initialized, Role: FOLLOWER
I20240502 08:17:00.481459 10110 consensus_queue.cc:260] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } }
I20240502 08:17:00.482720 10110 ts_tablet_manager.cc:1398] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 08:17:00.483618 8335 tablet_copy_service.cc:342] P 89ae15f3455446339957f9559f3857da: Request end of tablet copy session 6e2d5eaefade459c8e1fce0db41f2aca-aa41e5f5e3ef4dec89a0a39198f7a53c received from {username='slave'} at 127.6.5.7:47405
I20240502 08:17:00.483795 8335 tablet_copy_service.cc:433] P 89ae15f3455446339957f9559f3857da: ending tablet copy session 6e2d5eaefade459c8e1fce0db41f2aca-aa41e5f5e3ef4dec89a0a39198f7a53c on tablet aa41e5f5e3ef4dec89a0a39198f7a53c with peer 6e2d5eaefade459c8e1fce0db41f2aca
I20240502 08:17:00.992619 9925 raft_consensus.cc:1203] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:00.994408 10120 raft_consensus.cc:2932] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6e2d5eaefade459c8e1fce0db41f2aca (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } } }
I20240502 08:17:00.995222 9925 raft_consensus.cc:2932] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6e2d5eaefade459c8e1fce0db41f2aca (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } } }
I20240502 08:17:00.999853 8039 catalog_manager.cc:5484] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da reported cstate change: config changed from index -1 to 4, VOTER 6e2d5eaefade459c8e1fce0db41f2aca (127.6.5.7) added. New cstate: current_term: 1 leader_uuid: "89ae15f3455446339957f9559f3857da" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } } }
I20240502 08:17:01.028959 8185 consensus_queue.cc:237] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } }
W20240502 08:17:01.036728 8120 consensus_peers.cc:485] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 -> Peer 646203131e6c42a2a2e631374efbe5ee (127.6.5.6:34035): Couldn't send request to peer 646203131e6c42a2a2e631374efbe5ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f87d008812d847bb89dcf4034387bb51. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:01.640836 10149 ts_tablet_manager.cc:902] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee: Initiating tablet copy from peer b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:17:01.642396 10149 tablet_copy_client.cc:308] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:35591
I20240502 08:17:01.648319 8205 tablet_copy_service.cc:140] P b90db359bc744f2b82b69d97627fa7b9: Received BeginTabletCopySession request for tablet f87d008812d847bb89dcf4034387bb51 from peer 646203131e6c42a2a2e631374efbe5ee ({username='slave'} at 127.6.5.6:49037)
I20240502 08:17:01.648474 8205 tablet_copy_service.cc:161] P b90db359bc744f2b82b69d97627fa7b9: Beginning new tablet copy session on tablet f87d008812d847bb89dcf4034387bb51 from peer 646203131e6c42a2a2e631374efbe5ee at {username='slave'} at 127.6.5.6:49037: session id = 646203131e6c42a2a2e631374efbe5ee-f87d008812d847bb89dcf4034387bb51
I20240502 08:17:01.650348 8205 tablet_copy_source_session.cc:205] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:01.654242 10149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87d008812d847bb89dcf4034387bb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:01.664242 10149 tablet_copy_client.cc:783] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:01.664513 10149 tablet_copy_client.cc:647] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:01.666395 10149 tablet_copy_client.cc:523] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:01.669102 10149 tablet_bootstrap.cc:492] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee: Bootstrap starting.
I20240502 08:17:01.677290 10149 log.cc:824] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:01.706866 10149 tablet_bootstrap.cc:492] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:01.707330 10149 tablet_bootstrap.cc:492] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee: Bootstrap complete.
I20240502 08:17:01.707727 10149 ts_tablet_manager.cc:1367] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee: Time spent bootstrapping tablet: real 0.039s user 0.029s sys 0.010s
I20240502 08:17:01.711854 10149 raft_consensus.cc:348] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:17:01.712651 10149 raft_consensus.cc:729] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646203131e6c42a2a2e631374efbe5ee, State: Initialized, Role: FOLLOWER
I20240502 08:17:01.713039 10149 consensus_queue.cc:260] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } }
I20240502 08:17:01.714478 10149 ts_tablet_manager.cc:1398] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240502 08:17:01.715526 8205 tablet_copy_service.cc:342] P b90db359bc744f2b82b69d97627fa7b9: Request end of tablet copy session 646203131e6c42a2a2e631374efbe5ee-f87d008812d847bb89dcf4034387bb51 received from {username='slave'} at 127.6.5.6:49037
I20240502 08:17:01.715641 8205 tablet_copy_service.cc:433] P b90db359bc744f2b82b69d97627fa7b9: ending tablet copy session 646203131e6c42a2a2e631374efbe5ee-f87d008812d847bb89dcf4034387bb51 on tablet f87d008812d847bb89dcf4034387bb51 with peer 646203131e6c42a2a2e631374efbe5ee
I20240502 08:17:02.090263 9795 raft_consensus.cc:1203] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:02.091745 10158 raft_consensus.cc:2932] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 646203131e6c42a2a2e631374efbe5ee (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } } }
I20240502 08:17:02.092520 9795 raft_consensus.cc:2932] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 646203131e6c42a2a2e631374efbe5ee (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } } }
I20240502 08:17:02.094621 10158 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35224 (request call id 0) took 1066 ms. Trace:
I20240502 08:17:02.094813 10158 rpcz_store.cc:276] 0502 08:17:01.028482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:01.028529 (+ 47us) service_pool.cc:225] Handling call
0502 08:17:01.029536 (+ 1007us) log.cc:842] Serialized 248 byte log entry
0502 08:17:02.094561 (+1065025us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":63,"raft.queue_time_us":334,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":214,"thread_start_us":419,"threads_started":2,"wal-append.queue_time_us":270,"wal-append.run_cpu_time_us":485,"wal-append.run_wall_time_us":1000572}
I20240502 08:17:02.097846 8039 catalog_manager.cc:5484] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 reported cstate change: config changed from index -1 to 4, VOTER 646203131e6c42a2a2e631374efbe5ee (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } } }
I20240502 08:17:02.124006 8315 consensus_queue.cc:237] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } }
W20240502 08:17:02.131369 8248 consensus_peers.cc:485] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da -> Peer 00b1227597554ea18f7e125de6d2760e (127.6.5.3:43567): Couldn't send request to peer 00b1227597554ea18f7e125de6d2760e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fabef91cc4d4df3abfe3bdb393790e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:02.735795 10186 ts_tablet_manager.cc:902] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e: Initiating tablet copy from peer 89ae15f3455446339957f9559f3857da (127.6.5.2:44539)
I20240502 08:17:02.737525 10186 tablet_copy_client.cc:308] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:44539
I20240502 08:17:02.743853 8335 tablet_copy_service.cc:140] P 89ae15f3455446339957f9559f3857da: Received BeginTabletCopySession request for tablet 1fabef91cc4d4df3abfe3bdb393790e8 from peer 00b1227597554ea18f7e125de6d2760e ({username='slave'} at 127.6.5.3:50879)
I20240502 08:17:02.744011 8335 tablet_copy_service.cc:161] P 89ae15f3455446339957f9559f3857da: Beginning new tablet copy session on tablet 1fabef91cc4d4df3abfe3bdb393790e8 from peer 00b1227597554ea18f7e125de6d2760e at {username='slave'} at 127.6.5.3:50879: session id = 00b1227597554ea18f7e125de6d2760e-1fabef91cc4d4df3abfe3bdb393790e8
I20240502 08:17:02.745898 8335 tablet_copy_source_session.cc:205] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:02.749881 10186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fabef91cc4d4df3abfe3bdb393790e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.759785 10186 tablet_copy_client.cc:783] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:02.760049 10186 tablet_copy_client.cc:647] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:02.762012 10186 tablet_copy_client.cc:523] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:02.764901 10186 tablet_bootstrap.cc:492] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e: Bootstrap starting.
I20240502 08:17:02.773550 10186 log.cc:824] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:02.801671 10186 tablet_bootstrap.cc:492] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:02.802211 10186 tablet_bootstrap.cc:492] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e: Bootstrap complete.
I20240502 08:17:02.802631 10186 ts_tablet_manager.cc:1367] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e: Time spent bootstrapping tablet: real 0.038s user 0.026s sys 0.014s
I20240502 08:17:02.806805 10186 raft_consensus.cc:348] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:17:02.807621 10186 raft_consensus.cc:729] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b1227597554ea18f7e125de6d2760e, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.808029 10186 consensus_queue.cc:260] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } }
I20240502 08:17:02.809438 10186 ts_tablet_manager.cc:1398] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240502 08:17:02.810406 8335 tablet_copy_service.cc:342] P 89ae15f3455446339957f9559f3857da: Request end of tablet copy session 00b1227597554ea18f7e125de6d2760e-1fabef91cc4d4df3abfe3bdb393790e8 received from {username='slave'} at 127.6.5.3:50879
I20240502 08:17:02.810562 8335 tablet_copy_service.cc:433] P 89ae15f3455446339957f9559f3857da: ending tablet copy session 00b1227597554ea18f7e125de6d2760e-1fabef91cc4d4df3abfe3bdb393790e8 on tablet 1fabef91cc4d4df3abfe3bdb393790e8 with peer 00b1227597554ea18f7e125de6d2760e
I20240502 08:17:03.165886 9393 raft_consensus.cc:1203] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:03.167618 10195 raft_consensus.cc:2932] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 00b1227597554ea18f7e125de6d2760e (127.6.5.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } } }
I20240502 08:17:03.168246 9393 raft_consensus.cc:2932] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 00b1227597554ea18f7e125de6d2760e (127.6.5.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } } }
I20240502 08:17:03.170714 10195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34366 (request call id 0) took 1047 ms. Trace:
I20240502 08:17:03.170923 10195 rpcz_store.cc:276] 0502 08:17:02.123543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:02.123612 (+ 69us) service_pool.cc:225] Handling call
0502 08:17:02.124597 (+ 985us) log.cc:842] Serialized 248 byte log entry
0502 08:17:03.170668 (+1046071us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":54,"raft.queue_time_us":50,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":149,"thread_start_us":192,"threads_started":1,"wal-append.queue_time_us":265,"wal-append.run_cpu_time_us":372,"wal-append.run_wall_time_us":1000427}
I20240502 08:17:03.173048 8039 catalog_manager.cc:5484] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e reported cstate change: config changed from index -1 to 4, VOTER 00b1227597554ea18f7e125de6d2760e (127.6.5.3) added. New cstate: current_term: 1 leader_uuid: "89ae15f3455446339957f9559f3857da" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } } }
I20240502 08:17:03.206295 8185 consensus_queue.cc:237] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
W20240502 08:17:03.213665 8118 consensus_peers.cc:485] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 -> Peer b58bd494983a4a2885cda3387b1be902 (127.6.5.4:35529): Couldn't send request to peer b58bd494983a4a2885cda3387b1be902. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bdce98c845724dc3a9cedf89e3bec62c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:03.832427 10222 ts_tablet_manager.cc:902] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902: Initiating tablet copy from peer b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:17:03.834453 10222 tablet_copy_client.cc:308] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:35591
I20240502 08:17:03.840523 8205 tablet_copy_service.cc:140] P b90db359bc744f2b82b69d97627fa7b9: Received BeginTabletCopySession request for tablet bdce98c845724dc3a9cedf89e3bec62c from peer b58bd494983a4a2885cda3387b1be902 ({username='slave'} at 127.6.5.4:52557)
I20240502 08:17:03.840664 8205 tablet_copy_service.cc:161] P b90db359bc744f2b82b69d97627fa7b9: Beginning new tablet copy session on tablet bdce98c845724dc3a9cedf89e3bec62c from peer b58bd494983a4a2885cda3387b1be902 at {username='slave'} at 127.6.5.4:52557: session id = b58bd494983a4a2885cda3387b1be902-bdce98c845724dc3a9cedf89e3bec62c
I20240502 08:17:03.842607 8205 tablet_copy_source_session.cc:205] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:03.844746 10222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdce98c845724dc3a9cedf89e3bec62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:03.850433 10222 tablet_copy_client.cc:783] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:03.850731 10222 tablet_copy_client.cc:647] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:03.852535 10222 tablet_copy_client.cc:523] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:03.854960 10222 tablet_bootstrap.cc:492] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902: Bootstrap starting.
I20240502 08:17:03.887702 10222 tablet_bootstrap.cc:492] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:03.888058 10222 tablet_bootstrap.cc:492] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902: Bootstrap complete.
I20240502 08:17:03.888285 10222 ts_tablet_manager.cc:1367] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.004s
I20240502 08:17:03.888949 10222 raft_consensus.cc:348] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:17:03.889398 10222 raft_consensus.cc:729] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b58bd494983a4a2885cda3387b1be902, State: Initialized, Role: FOLLOWER
I20240502 08:17:03.889587 10222 consensus_queue.cc:260] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:17:03.890728 10222 ts_tablet_manager.cc:1398] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240502 08:17:03.891590 8205 tablet_copy_service.cc:342] P b90db359bc744f2b82b69d97627fa7b9: Request end of tablet copy session b58bd494983a4a2885cda3387b1be902-bdce98c845724dc3a9cedf89e3bec62c received from {username='slave'} at 127.6.5.4:52557
I20240502 08:17:03.891745 8205 tablet_copy_service.cc:433] P b90db359bc744f2b82b69d97627fa7b9: ending tablet copy session b58bd494983a4a2885cda3387b1be902-bdce98c845724dc3a9cedf89e3bec62c on tablet bdce98c845724dc3a9cedf89e3bec62c with peer b58bd494983a4a2885cda3387b1be902
I20240502 08:17:04.431902 9526 raft_consensus.cc:1203] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:04.433552 10232 raft_consensus.cc:2932] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b58bd494983a4a2885cda3387b1be902 (127.6.5.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } } }
I20240502 08:17:04.434774 9526 raft_consensus.cc:2932] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b58bd494983a4a2885cda3387b1be902 (127.6.5.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } } }
I20240502 08:17:04.437209 10232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35244 (request call id 0) took 1231 ms. Trace:
I20240502 08:17:04.437419 10232 rpcz_store.cc:276] 0502 08:17:03.205790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:03.205860 (+ 70us) service_pool.cc:225] Handling call
0502 08:17:03.206939 (+ 1079us) log.cc:842] Serialized 248 byte log entry
0502 08:17:04.437160 (+1230221us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":57,"raft.queue_time_us":50,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":173,"thread_start_us":215,"threads_started":1,"wal-append.queue_time_us":303,"wal-append.run_cpu_time_us":536,"wal-append.run_wall_time_us":1000598}
I20240502 08:17:04.439954 8039 catalog_manager.cc:5484] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 reported cstate change: config changed from index -1 to 4, VOTER b58bd494983a4a2885cda3387b1be902 (127.6.5.4) added. New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } } }
I20240502 08:17:04.467644 8315 consensus_queue.cc:237] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } }
W20240502 08:17:04.474653 8249 consensus_peers.cc:485] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da -> Peer c4e97cd51e9949339f937b6be4cefebb (127.6.5.5:34875): Couldn't send request to peer c4e97cd51e9949339f937b6be4cefebb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74a88bd56f86453280555608afcb98f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:04.961390 10259 ts_tablet_manager.cc:902] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb: Initiating tablet copy from peer 89ae15f3455446339957f9559f3857da (127.6.5.2:44539)
I20240502 08:17:04.962612 10259 tablet_copy_client.cc:308] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:44539
I20240502 08:17:04.969830 8335 tablet_copy_service.cc:140] P 89ae15f3455446339957f9559f3857da: Received BeginTabletCopySession request for tablet 74a88bd56f86453280555608afcb98f3 from peer c4e97cd51e9949339f937b6be4cefebb ({username='slave'} at 127.6.5.5:38185)
I20240502 08:17:04.969990 8335 tablet_copy_service.cc:161] P 89ae15f3455446339957f9559f3857da: Beginning new tablet copy session on tablet 74a88bd56f86453280555608afcb98f3 from peer c4e97cd51e9949339f937b6be4cefebb at {username='slave'} at 127.6.5.5:38185: session id = c4e97cd51e9949339f937b6be4cefebb-74a88bd56f86453280555608afcb98f3
I20240502 08:17:04.972059 8335 tablet_copy_source_session.cc:205] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:04.973331 10259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a88bd56f86453280555608afcb98f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.978312 10259 tablet_copy_client.cc:783] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:04.978541 10259 tablet_copy_client.cc:647] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:04.979899 10259 tablet_copy_client.cc:523] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:04.982009 10259 tablet_bootstrap.cc:492] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb: Bootstrap starting.
I20240502 08:17:05.009122 10259 tablet_bootstrap.cc:492] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:05.009464 10259 tablet_bootstrap.cc:492] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb: Bootstrap complete.
I20240502 08:17:05.009667 10259 ts_tablet_manager.cc:1367] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.004s
I20240502 08:17:05.010383 10259 raft_consensus.cc:348] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:17:05.010733 10259 raft_consensus.cc:729] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e97cd51e9949339f937b6be4cefebb, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.010892 10259 consensus_queue.cc:260] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } }
I20240502 08:17:05.011507 10259 ts_tablet_manager.cc:1398] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 08:17:05.012166 8335 tablet_copy_service.cc:342] P 89ae15f3455446339957f9559f3857da: Request end of tablet copy session c4e97cd51e9949339f937b6be4cefebb-74a88bd56f86453280555608afcb98f3 received from {username='slave'} at 127.6.5.5:38185
I20240502 08:17:05.012290 8335 tablet_copy_service.cc:433] P 89ae15f3455446339957f9559f3857da: ending tablet copy session c4e97cd51e9949339f937b6be4cefebb-74a88bd56f86453280555608afcb98f3 on tablet 74a88bd56f86453280555608afcb98f3 with peer c4e97cd51e9949339f937b6be4cefebb
I20240502 08:17:05.530668 9665 raft_consensus.cc:1203] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:05.532056 10268 raft_consensus.cc:2932] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c4e97cd51e9949339f937b6be4cefebb (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } } }
I20240502 08:17:05.533217 9665 raft_consensus.cc:2932] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c4e97cd51e9949339f937b6be4cefebb (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } } }
I20240502 08:17:05.535231 10268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34394 (request call id 0) took 1068 ms. Trace:
I20240502 08:17:05.535418 10268 rpcz_store.cc:276] 0502 08:17:04.467175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:04.467232 (+ 57us) service_pool.cc:225] Handling call
0502 08:17:04.468267 (+ 1035us) log.cc:842] Serialized 248 byte log entry
0502 08:17:05.535178 (+1066911us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":64,"raft.queue_time_us":41,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":186,"spinlock_wait_cycles":34560,"thread_start_us":176,"threads_started":1,"wal-append.queue_time_us":261,"wal-append.run_cpu_time_us":431,"wal-append.run_wall_time_us":1000500}
I20240502 08:17:05.538815 8039 catalog_manager.cc:5484] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb reported cstate change: config changed from index -1 to 4, VOTER c4e97cd51e9949339f937b6be4cefebb (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "89ae15f3455446339957f9559f3857da" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } } }
I20240502 08:17:05.564765 8185 consensus_queue.cc:237] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } }
W20240502 08:17:05.572024 8119 consensus_peers.cc:485] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 -> Peer 6e2d5eaefade459c8e1fce0db41f2aca (127.6.5.7:39789): Couldn't send request to peer 6e2d5eaefade459c8e1fce0db41f2aca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afc5bc8efa7d4a3fb8970bf100c754a2. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:06.155048 10295 ts_tablet_manager.cc:902] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca: Initiating tablet copy from peer b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:17:06.156100 10295 tablet_copy_client.cc:308] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:35591
I20240502 08:17:06.161453 8205 tablet_copy_service.cc:140] P b90db359bc744f2b82b69d97627fa7b9: Received BeginTabletCopySession request for tablet afc5bc8efa7d4a3fb8970bf100c754a2 from peer 6e2d5eaefade459c8e1fce0db41f2aca ({username='slave'} at 127.6.5.7:56787)
I20240502 08:17:06.161624 8205 tablet_copy_service.cc:161] P b90db359bc744f2b82b69d97627fa7b9: Beginning new tablet copy session on tablet afc5bc8efa7d4a3fb8970bf100c754a2 from peer 6e2d5eaefade459c8e1fce0db41f2aca at {username='slave'} at 127.6.5.7:56787: session id = 6e2d5eaefade459c8e1fce0db41f2aca-afc5bc8efa7d4a3fb8970bf100c754a2
I20240502 08:17:06.163992 8205 tablet_copy_source_session.cc:205] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:06.165179 10295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc5bc8efa7d4a3fb8970bf100c754a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:06.169308 10295 tablet_copy_client.cc:783] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:06.169503 10295 tablet_copy_client.cc:647] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:06.170953 10295 tablet_copy_client.cc:523] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:06.173085 10295 tablet_bootstrap.cc:492] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca: Bootstrap starting.
I20240502 08:17:06.202567 10295 tablet_bootstrap.cc:492] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:06.202896 10295 tablet_bootstrap.cc:492] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca: Bootstrap complete.
I20240502 08:17:06.203117 10295 ts_tablet_manager.cc:1367] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca: Time spent bootstrapping tablet: real 0.030s user 0.030s sys 0.002s
I20240502 08:17:06.203853 10295 raft_consensus.cc:348] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:17:06.204200 10295 raft_consensus.cc:729] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2d5eaefade459c8e1fce0db41f2aca, State: Initialized, Role: FOLLOWER
I20240502 08:17:06.204368 10295 consensus_queue.cc:260] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } }
I20240502 08:17:06.205040 10295 ts_tablet_manager.cc:1398] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:06.205711 8205 tablet_copy_service.cc:342] P b90db359bc744f2b82b69d97627fa7b9: Request end of tablet copy session 6e2d5eaefade459c8e1fce0db41f2aca-afc5bc8efa7d4a3fb8970bf100c754a2 received from {username='slave'} at 127.6.5.7:56787
I20240502 08:17:06.205818 8205 tablet_copy_service.cc:433] P b90db359bc744f2b82b69d97627fa7b9: ending tablet copy session 6e2d5eaefade459c8e1fce0db41f2aca-afc5bc8efa7d4a3fb8970bf100c754a2 on tablet afc5bc8efa7d4a3fb8970bf100c754a2 with peer 6e2d5eaefade459c8e1fce0db41f2aca
I20240502 08:17:06.601064 9925 raft_consensus.cc:1203] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:06.602250 10304 raft_consensus.cc:2932] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6e2d5eaefade459c8e1fce0db41f2aca (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } } }
I20240502 08:17:06.602746 9925 raft_consensus.cc:2932] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6e2d5eaefade459c8e1fce0db41f2aca (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } } }
I20240502 08:17:06.605242 10304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35266 (request call id 0) took 1041 ms. Trace:
I20240502 08:17:06.605381 10304 rpcz_store.cc:276] 0502 08:17:05.564216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:05.564285 (+ 69us) service_pool.cc:225] Handling call
0502 08:17:05.565382 (+ 1097us) log.cc:842] Serialized 248 byte log entry
0502 08:17:06.605190 (+1039808us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":56,"raft.queue_time_us":46,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":223,"spinlock_wait_cycles":26880,"thread_start_us":176,"threads_started":1,"wal-append.queue_time_us":291,"wal-append.run_cpu_time_us":455,"wal-append.run_wall_time_us":1000518}
I20240502 08:17:06.608708 8041 catalog_manager.cc:5484] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca reported cstate change: config changed from index -1 to 4, VOTER 6e2d5eaefade459c8e1fce0db41f2aca (127.6.5.7) added. New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } } }
I20240502 08:17:06.637257 8315 consensus_queue.cc:237] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } }
W20240502 08:17:06.644268 8250 consensus_peers.cc:485] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da -> Peer 646203131e6c42a2a2e631374efbe5ee (127.6.5.6:34035): Couldn't send request to peer 646203131e6c42a2a2e631374efbe5ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d96e8d4b891410980ae6a696799a60b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:07.018860 10331 ts_tablet_manager.cc:902] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee: Initiating tablet copy from peer 89ae15f3455446339957f9559f3857da (127.6.5.2:44539)
I20240502 08:17:07.020409 10331 tablet_copy_client.cc:308] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:44539
I20240502 08:17:07.026043 8335 tablet_copy_service.cc:140] P 89ae15f3455446339957f9559f3857da: Received BeginTabletCopySession request for tablet 3d96e8d4b891410980ae6a696799a60b from peer 646203131e6c42a2a2e631374efbe5ee ({username='slave'} at 127.6.5.6:49739)
I20240502 08:17:07.026194 8335 tablet_copy_service.cc:161] P 89ae15f3455446339957f9559f3857da: Beginning new tablet copy session on tablet 3d96e8d4b891410980ae6a696799a60b from peer 646203131e6c42a2a2e631374efbe5ee at {username='slave'} at 127.6.5.6:49739: session id = 646203131e6c42a2a2e631374efbe5ee-3d96e8d4b891410980ae6a696799a60b
I20240502 08:17:07.028172 8335 tablet_copy_source_session.cc:205] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:07.029291 10331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d96e8d4b891410980ae6a696799a60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:07.033463 10331 tablet_copy_client.cc:783] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:07.033710 10331 tablet_copy_client.cc:647] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:07.035241 10331 tablet_copy_client.cc:523] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:07.037340 10331 tablet_bootstrap.cc:492] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee: Bootstrap starting.
I20240502 08:17:07.064844 10331 tablet_bootstrap.cc:492] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:07.065219 10331 tablet_bootstrap.cc:492] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee: Bootstrap complete.
I20240502 08:17:07.065423 10331 ts_tablet_manager.cc:1367] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.009s
I20240502 08:17:07.066145 10331 raft_consensus.cc:348] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:17:07.066675 10331 raft_consensus.cc:729] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646203131e6c42a2a2e631374efbe5ee, State: Initialized, Role: FOLLOWER
I20240502 08:17:07.066829 10331 consensus_queue.cc:260] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } }
I20240502 08:17:07.067396 10331 ts_tablet_manager.cc:1398] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 08:17:07.068238 8335 tablet_copy_service.cc:342] P 89ae15f3455446339957f9559f3857da: Request end of tablet copy session 646203131e6c42a2a2e631374efbe5ee-3d96e8d4b891410980ae6a696799a60b received from {username='slave'} at 127.6.5.6:49739
I20240502 08:17:07.068343 8335 tablet_copy_service.cc:433] P 89ae15f3455446339957f9559f3857da: ending tablet copy session 646203131e6c42a2a2e631374efbe5ee-3d96e8d4b891410980ae6a696799a60b on tablet 3d96e8d4b891410980ae6a696799a60b with peer 646203131e6c42a2a2e631374efbe5ee
I20240502 08:17:07.448897 9795 raft_consensus.cc:1203] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:07.450320 10340 raft_consensus.cc:2932] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 646203131e6c42a2a2e631374efbe5ee (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } } }
I20240502 08:17:07.450986 9795 raft_consensus.cc:2932] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 646203131e6c42a2a2e631374efbe5ee (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } } }
I20240502 08:17:07.455242 8040 catalog_manager.cc:5484] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da reported cstate change: config changed from index -1 to 4, VOTER 646203131e6c42a2a2e631374efbe5ee (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "89ae15f3455446339957f9559f3857da" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } } }
I20240502 08:17:07.487639 8185 consensus_queue.cc:237] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } }
W20240502 08:17:07.495086 8118 consensus_peers.cc:485] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 -> Peer 00b1227597554ea18f7e125de6d2760e (127.6.5.3:43567): Couldn't send request to peer 00b1227597554ea18f7e125de6d2760e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b3e2b0c939e4182bb851b4cdc802301. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:08.077483 10365 ts_tablet_manager.cc:902] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e: Initiating tablet copy from peer b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:17:08.078575 10365 tablet_copy_client.cc:308] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:35591
I20240502 08:17:08.084195 8205 tablet_copy_service.cc:140] P b90db359bc744f2b82b69d97627fa7b9: Received BeginTabletCopySession request for tablet 0b3e2b0c939e4182bb851b4cdc802301 from peer 00b1227597554ea18f7e125de6d2760e ({username='slave'} at 127.6.5.3:46601)
I20240502 08:17:08.084338 8205 tablet_copy_service.cc:161] P b90db359bc744f2b82b69d97627fa7b9: Beginning new tablet copy session on tablet 0b3e2b0c939e4182bb851b4cdc802301 from peer 00b1227597554ea18f7e125de6d2760e at {username='slave'} at 127.6.5.3:46601: session id = 00b1227597554ea18f7e125de6d2760e-0b3e2b0c939e4182bb851b4cdc802301
I20240502 08:17:08.086299 8205 tablet_copy_source_session.cc:205] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:08.087652 10365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3e2b0c939e4182bb851b4cdc802301. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:08.092110 10365 tablet_copy_client.cc:783] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:08.092317 10365 tablet_copy_client.cc:647] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:08.093868 10365 tablet_copy_client.cc:523] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:08.096105 10365 tablet_bootstrap.cc:492] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e: Bootstrap starting.
I20240502 08:17:08.124895 10365 tablet_bootstrap.cc:492] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:08.125239 10365 tablet_bootstrap.cc:492] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e: Bootstrap complete.
I20240502 08:17:08.125487 10365 ts_tablet_manager.cc:1367] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.001s
I20240502 08:17:08.126268 10365 raft_consensus.cc:348] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:17:08.126648 10365 raft_consensus.cc:729] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b1227597554ea18f7e125de6d2760e, State: Initialized, Role: FOLLOWER
I20240502 08:17:08.126812 10365 consensus_queue.cc:260] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } }
I20240502 08:17:08.127429 10365 ts_tablet_manager.cc:1398] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 08:17:08.128253 8205 tablet_copy_service.cc:342] P b90db359bc744f2b82b69d97627fa7b9: Request end of tablet copy session 00b1227597554ea18f7e125de6d2760e-0b3e2b0c939e4182bb851b4cdc802301 received from {username='slave'} at 127.6.5.3:46601
I20240502 08:17:08.128383 8205 tablet_copy_service.cc:433] P b90db359bc744f2b82b69d97627fa7b9: ending tablet copy session 00b1227597554ea18f7e125de6d2760e-0b3e2b0c939e4182bb851b4cdc802301 on tablet 0b3e2b0c939e4182bb851b4cdc802301 with peer 00b1227597554ea18f7e125de6d2760e
I20240502 08:17:08.550388 9393 raft_consensus.cc:1203] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:08.551604 10374 raft_consensus.cc:2932] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 00b1227597554ea18f7e125de6d2760e (127.6.5.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } } }
I20240502 08:17:08.552232 9393 raft_consensus.cc:2932] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 00b1227597554ea18f7e125de6d2760e (127.6.5.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } } }
I20240502 08:17:08.555229 10374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35292 (request call id 0) took 1068 ms. Trace:
I20240502 08:17:08.555438 10374 rpcz_store.cc:276] 0502 08:17:07.487150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:07.487215 (+ 65us) service_pool.cc:225] Handling call
0502 08:17:07.488248 (+ 1033us) log.cc:842] Serialized 248 byte log entry
0502 08:17:08.555179 (+1066931us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":55,"raft.queue_time_us":51,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":163,"spinlock_wait_cycles":33536,"thread_start_us":172,"threads_started":1,"wal-append.queue_time_us":265,"wal-append.run_cpu_time_us":493,"wal-append.run_wall_time_us":1000548}
I20240502 08:17:08.557524 8040 catalog_manager.cc:5484] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e reported cstate change: config changed from index -1 to 4, VOTER 00b1227597554ea18f7e125de6d2760e (127.6.5.3) added. New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } } }
I20240502 08:17:08.660429 9775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:08.664652 9373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:08.680076 9905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:08.692103 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:08.717010 8295 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:17:08.723980 8165 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:17:08.754984 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:08.795301 8185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0b3e2b0c939e4182bb851b4cdc802301"
dest_uuid: "b90db359bc744f2b82b69d97627fa7b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35306
I20240502 08:17:08.795565 8185 raft_consensus.cc:595] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:08.889751 9905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:08.899224 9373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:08.905630 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:08.910941 9775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:08.911342 8295 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:17:08.950428 8165 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:17:08.962152 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.103407 9373 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.106220 9775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.114917 8295 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:17:09.129150 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:09.155992 10498 raft_consensus.cc:979] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9: : Instructing follower 00b1227597554ea18f7e125de6d2760e to start an election
I20240502 08:17:09.156230 10498 raft_consensus.cc:1067] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Signalling peer 00b1227597554ea18f7e125de6d2760e to start an election
I20240502 08:17:09.161096 9393 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0b3e2b0c939e4182bb851b4cdc802301"
dest_uuid: "00b1227597554ea18f7e125de6d2760e"
from {username='slave'} at 127.6.5.1:47291
I20240502 08:17:09.161398 9393 raft_consensus.cc:482] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:09.161628 9393 raft_consensus.cc:3037] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:09.162735 8165 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:17:09.165076 9393 raft_consensus.cc:504] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } }
I20240502 08:17:09.167467 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3e2b0c939e4182bb851b4cdc802301" candidate_uuid: "00b1227597554ea18f7e125de6d2760e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b90db359bc744f2b82b69d97627fa7b9"
I20240502 08:17:09.167812 8185 raft_consensus.cc:3032] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:09.167903 8185 raft_consensus.cc:729] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:17:09.168224 8185 consensus_queue.cc:260] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } }
I20240502 08:17:09.169759 8185 raft_consensus.cc:3037] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:09.172194 9905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.172928 8185 raft_consensus.cc:2454] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b1227597554ea18f7e125de6d2760e in term 2.
I20240502 08:17:09.173538 9329 leader_election.cc:302] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 00b1227597554ea18f7e125de6d2760e, b90db359bc744f2b82b69d97627fa7b9; no voters:
I20240502 08:17:09.176054 9393 leader_election.cc:288] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [CANDIDATE]: Term 2 election: Requested vote from peers b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:17:09.189195 10500 raft_consensus.cc:2781] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:09.193804 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.195796 10500 raft_consensus.cc:686] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [term 2 LEADER]: Becoming Leader. State: Replica: 00b1227597554ea18f7e125de6d2760e, State: Running, Role: LEADER
I20240502 08:17:09.196360 10500 consensus_queue.cc:237] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } }
I20240502 08:17:09.199501 8043 catalog_manager.cc:5484] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e reported cstate change: term changed from 1 to 2, leader changed from b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) to 00b1227597554ea18f7e125de6d2760e (127.6.5.3). New cstate: current_term: 2 leader_uuid: "00b1227597554ea18f7e125de6d2760e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } } }
I20240502 08:17:09.233029 8315 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d96e8d4b891410980ae6a696799a60b"
dest_uuid: "89ae15f3455446339957f9559f3857da"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34452
I20240502 08:17:09.233264 8315 raft_consensus.cc:595] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:09.326555 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:09.334441 9775 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.337096 9905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.348529 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:09.356679 8295 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:17:09.373335 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.392367 8165 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:17:09.448194 10551 raft_consensus.cc:979] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da: : Instructing follower 646203131e6c42a2a2e631374efbe5ee to start an election
I20240502 08:17:09.448487 10340 raft_consensus.cc:1067] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Signalling peer 646203131e6c42a2a2e631374efbe5ee to start an election
I20240502 08:17:09.449887 9795 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d96e8d4b891410980ae6a696799a60b"
dest_uuid: "646203131e6c42a2a2e631374efbe5ee"
from {username='slave'} at 127.6.5.2:43323
I20240502 08:17:09.450168 9795 raft_consensus.cc:482] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:09.450350 9795 raft_consensus.cc:3037] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:09.452755 9795 raft_consensus.cc:504] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } }
I20240502 08:17:09.454156 9795 leader_election.cc:288] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [CANDIDATE]: Term 2 election: Requested vote from peers 89ae15f3455446339957f9559f3857da (127.6.5.2:44539)
I20240502 08:17:09.454802 8315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d96e8d4b891410980ae6a696799a60b" candidate_uuid: "646203131e6c42a2a2e631374efbe5ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "89ae15f3455446339957f9559f3857da"
I20240502 08:17:09.455068 8315 raft_consensus.cc:3032] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:09.455193 8315 raft_consensus.cc:729] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 89ae15f3455446339957f9559f3857da, State: Running, Role: LEADER
I20240502 08:17:09.455464 8315 consensus_queue.cc:260] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } }
I20240502 08:17:09.455861 8315 raft_consensus.cc:3037] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:09.457787 8315 raft_consensus.cc:2454] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 646203131e6c42a2a2e631374efbe5ee in term 2.
I20240502 08:17:09.458297 9731 leader_election.cc:302] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 646203131e6c42a2a2e631374efbe5ee, 89ae15f3455446339957f9559f3857da; no voters:
I20240502 08:17:09.458935 10553 raft_consensus.cc:2781] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:09.459832 10553 raft_consensus.cc:686] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [term 2 LEADER]: Becoming Leader. State: Replica: 646203131e6c42a2a2e631374efbe5ee, State: Running, Role: LEADER
I20240502 08:17:09.460347 10553 consensus_queue.cc:237] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } }
I20240502 08:17:09.463968 8039 catalog_manager.cc:5484] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee reported cstate change: term changed from 1 to 2, leader changed from 89ae15f3455446339957f9559f3857da (127.6.5.2) to 646203131e6c42a2a2e631374efbe5ee (127.6.5.6). New cstate: current_term: 2 leader_uuid: "646203131e6c42a2a2e631374efbe5ee" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } } }
I20240502 08:17:09.532018 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:09.535094 9775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:09.549198 9905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.556677 8295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:17:09.565142 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:09.580657 8165 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:17:09.603494 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.649375 8185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "afc5bc8efa7d4a3fb8970bf100c754a2"
dest_uuid: "b90db359bc744f2b82b69d97627fa7b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35360
I20240502 08:17:09.649585 8185 raft_consensus.cc:595] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:09.710201 8185 raft_consensus.cc:1261] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Refusing update from remote peer 00b1227597554ea18f7e125de6d2760e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:17:09.711181 10609 consensus_queue.cc:1035] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:09.755142 9775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:09.760985 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:09.778666 8295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:17:09.780601 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:09.793469 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.793999 8165 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:17:09.795917 9905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.962328 9905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:09.970170 9775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:09.973652 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:09.981860 8295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:17:10.001317 8165 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:17:10.012208 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.015213 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:10.027735 8315 raft_consensus.cc:1261] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Refusing update from remote peer 646203131e6c42a2a2e631374efbe5ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:17:10.033442 10498 raft_consensus.cc:979] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9: : Instructing follower 6e2d5eaefade459c8e1fce0db41f2aca to start an election
I20240502 08:17:10.035952 10664 raft_consensus.cc:1067] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Signalling peer 6e2d5eaefade459c8e1fce0db41f2aca to start an election
I20240502 08:17:10.037168 9925 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "afc5bc8efa7d4a3fb8970bf100c754a2"
dest_uuid: "6e2d5eaefade459c8e1fce0db41f2aca"
from {username='slave'} at 127.6.5.1:57473
I20240502 08:17:10.037456 9925 raft_consensus.cc:482] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:10.037624 9925 raft_consensus.cc:3037] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:10.039007 10662 consensus_queue.cc:1035] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:10.040470 9925 raft_consensus.cc:504] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } }
I20240502 08:17:10.042650 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc5bc8efa7d4a3fb8970bf100c754a2" candidate_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b90db359bc744f2b82b69d97627fa7b9"
I20240502 08:17:10.042995 9925 leader_election.cc:288] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [CANDIDATE]: Term 2 election: Requested vote from peers b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:17:10.051683 8185 raft_consensus.cc:3032] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:10.051791 8185 raft_consensus.cc:729] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:17:10.052021 8185 consensus_queue.cc:260] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } }
I20240502 08:17:10.052340 8185 raft_consensus.cc:3037] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:10.055397 8185 raft_consensus.cc:2454] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2d5eaefade459c8e1fce0db41f2aca in term 2.
I20240502 08:17:10.056355 9861 leader_election.cc:302] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6e2d5eaefade459c8e1fce0db41f2aca, b90db359bc744f2b82b69d97627fa7b9; no voters:
I20240502 08:17:10.066288 10668 raft_consensus.cc:2781] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:10.074637 10668 raft_consensus.cc:686] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [term 2 LEADER]: Becoming Leader. State: Replica: 6e2d5eaefade459c8e1fce0db41f2aca, State: Running, Role: LEADER
I20240502 08:17:10.075345 10668 consensus_queue.cc:237] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } }
I20240502 08:17:10.079434 8040 catalog_manager.cc:5484] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca reported cstate change: term changed from 1 to 2, leader changed from b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) to 6e2d5eaefade459c8e1fce0db41f2aca (127.6.5.7). New cstate: current_term: 2 leader_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } } }
I20240502 08:17:10.099560 8314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa41e5f5e3ef4dec89a0a39198f7a53c"
dest_uuid: "89ae15f3455446339957f9559f3857da"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51992
I20240502 08:17:10.099726 8314 raft_consensus.cc:595] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:10.189195 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.193387 9775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.196889 9905 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.197850 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.227756 8295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:17:10.231376 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:10.253041 8165 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:17:10.408116 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.418634 9775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.430506 8295 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:17:10.439411 9905 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.449290 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:10.449927 8165 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:17:10.455004 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.526933 8185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "72b8cb6631624a839ec2b3a77abaa1df"
dest_uuid: "b90db359bc744f2b82b69d97627fa7b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60234
I20240502 08:17:10.527135 8185 raft_consensus.cc:595] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:10.530850 8185 raft_consensus.cc:1261] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Refusing update from remote peer 6e2d5eaefade459c8e1fce0db41f2aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:17:10.531517 10668 consensus_queue.cc:1035] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:10.589242 10778 raft_consensus.cc:979] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da: : Instructing follower 6e2d5eaefade459c8e1fce0db41f2aca to start an election
I20240502 08:17:10.592186 10340 raft_consensus.cc:1067] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Signalling peer 6e2d5eaefade459c8e1fce0db41f2aca to start an election
I20240502 08:17:10.593070 9925 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa41e5f5e3ef4dec89a0a39198f7a53c"
dest_uuid: "6e2d5eaefade459c8e1fce0db41f2aca"
from {username='slave'} at 127.6.5.2:37219
I20240502 08:17:10.593304 9925 raft_consensus.cc:482] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:10.593423 9925 raft_consensus.cc:3037] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:10.595773 9925 raft_consensus.cc:504] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } }
I20240502 08:17:10.596992 8314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa41e5f5e3ef4dec89a0a39198f7a53c" candidate_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "89ae15f3455446339957f9559f3857da"
I20240502 08:17:10.597214 8314 raft_consensus.cc:3032] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:10.597306 8314 raft_consensus.cc:729] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 89ae15f3455446339957f9559f3857da, State: Running, Role: LEADER
I20240502 08:17:10.597565 8314 consensus_queue.cc:260] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } }
I20240502 08:17:10.597765 8314 raft_consensus.cc:3037] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:10.600247 8314 raft_consensus.cc:2454] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2d5eaefade459c8e1fce0db41f2aca in term 2.
I20240502 08:17:10.600488 9925 leader_election.cc:288] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [CANDIDATE]: Term 2 election: Requested vote from peers 89ae15f3455446339957f9559f3857da (127.6.5.2:44539)
I20240502 08:17:10.600950 9861 leader_election.cc:302] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6e2d5eaefade459c8e1fce0db41f2aca, 89ae15f3455446339957f9559f3857da; no voters:
I20240502 08:17:10.601253 10669 raft_consensus.cc:2781] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:10.601383 10669 raft_consensus.cc:686] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [term 2 LEADER]: Becoming Leader. State: Replica: 6e2d5eaefade459c8e1fce0db41f2aca, State: Running, Role: LEADER
I20240502 08:17:10.601655 10669 consensus_queue.cc:237] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } }
I20240502 08:17:10.604202 8043 catalog_manager.cc:5484] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca reported cstate change: term changed from 1 to 2, leader changed from 89ae15f3455446339957f9559f3857da (127.6.5.2) to 6e2d5eaefade459c8e1fce0db41f2aca (127.6.5.7). New cstate: current_term: 2 leader_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } } }
I20240502 08:17:10.617648 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.630028 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:10.635454 8295 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:10.646883 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.658713 9775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.665751 8165 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:17:10.679124 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:10.842195 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.852193 9775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.854597 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:10.868805 9645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:10.870297 9505 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:10.909574 8295 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:10.910205 8165 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:17:10.958559 8185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bdce98c845724dc3a9cedf89e3bec62c"
dest_uuid: "b90db359bc744f2b82b69d97627fa7b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60268
I20240502 08:17:10.958763 8185 raft_consensus.cc:595] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:10.985664 10843 raft_consensus.cc:979] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9: : Instructing follower b58bd494983a4a2885cda3387b1be902 to start an election
I20240502 08:17:10.985829 10843 raft_consensus.cc:1067] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Signalling peer b58bd494983a4a2885cda3387b1be902 to start an election
I20240502 08:17:10.987032 9526 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdce98c845724dc3a9cedf89e3bec62c"
dest_uuid: "b58bd494983a4a2885cda3387b1be902"
from {username='slave'} at 127.6.5.1:39465
I20240502 08:17:10.987316 9526 raft_consensus.cc:482] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:10.987449 9526 raft_consensus.cc:3037] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:10.989863 9526 raft_consensus.cc:504] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:17:10.990826 9526 leader_election.cc:288] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [CANDIDATE]: Term 2 election: Requested vote from peers b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:17:10.991494 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdce98c845724dc3a9cedf89e3bec62c" candidate_uuid: "b58bd494983a4a2885cda3387b1be902" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b90db359bc744f2b82b69d97627fa7b9"
I20240502 08:17:10.992108 8185 raft_consensus.cc:3032] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:10.992189 8185 raft_consensus.cc:729] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:17:10.992419 8185 consensus_queue.cc:260] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:17:10.992660 8185 raft_consensus.cc:3037] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:10.994505 8185 raft_consensus.cc:2454] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b58bd494983a4a2885cda3387b1be902 in term 2.
I20240502 08:17:10.995034 9460 leader_election.cc:302] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b58bd494983a4a2885cda3387b1be902, b90db359bc744f2b82b69d97627fa7b9; no voters:
I20240502 08:17:10.995635 10844 raft_consensus.cc:2781] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:10.996007 10844 raft_consensus.cc:686] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [term 2 LEADER]: Becoming Leader. State: Replica: b58bd494983a4a2885cda3387b1be902, State: Running, Role: LEADER
I20240502 08:17:10.996342 10844 consensus_queue.cc:237] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:17:10.998952 8038 catalog_manager.cc:5484] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 reported cstate change: term changed from 1 to 2, leader changed from b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) to b58bd494983a4a2885cda3387b1be902 (127.6.5.4). New cstate: current_term: 2 leader_uuid: "b58bd494983a4a2885cda3387b1be902" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } } }
I20240502 08:17:11.017370 10664 raft_consensus.cc:979] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9: : Instructing follower c4e97cd51e9949339f937b6be4cefebb to start an election
I20240502 08:17:11.017545 10664 raft_consensus.cc:1067] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Signalling peer c4e97cd51e9949339f937b6be4cefebb to start an election
I20240502 08:17:11.018234 9665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72b8cb6631624a839ec2b3a77abaa1df"
dest_uuid: "c4e97cd51e9949339f937b6be4cefebb"
from {username='slave'} at 127.6.5.1:58265
I20240502 08:17:11.018487 9665 raft_consensus.cc:482] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:11.018700 9665 raft_consensus.cc:3037] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:11.021184 9665 raft_consensus.cc:504] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } }
I20240502 08:17:11.023324 9665 leader_election.cc:288] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [CANDIDATE]: Term 2 election: Requested vote from peers b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:17:11.024226 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8cb6631624a839ec2b3a77abaa1df" candidate_uuid: "c4e97cd51e9949339f937b6be4cefebb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b90db359bc744f2b82b69d97627fa7b9"
I20240502 08:17:11.024447 8185 raft_consensus.cc:3032] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:11.024525 8185 raft_consensus.cc:729] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:17:11.024771 8185 consensus_queue.cc:260] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } }
I20240502 08:17:11.025430 8185 raft_consensus.cc:3037] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:11.026640 8314 raft_consensus.cc:1261] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Refusing update from remote peer 6e2d5eaefade459c8e1fce0db41f2aca: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:17:11.027211 10669 consensus_queue.cc:1035] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:11.027534 8185 raft_consensus.cc:2454] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4e97cd51e9949339f937b6be4cefebb in term 2.
I20240502 08:17:11.028071 9601 leader_election.cc:302] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b90db359bc744f2b82b69d97627fa7b9, c4e97cd51e9949339f937b6be4cefebb; no voters:
I20240502 08:17:11.028725 10862 raft_consensus.cc:2781] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:11.029420 10862 raft_consensus.cc:686] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [term 2 LEADER]: Becoming Leader. State: Replica: c4e97cd51e9949339f937b6be4cefebb, State: Running, Role: LEADER
I20240502 08:17:11.030048 10862 consensus_queue.cc:237] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } }
I20240502 08:17:11.034893 8038 catalog_manager.cc:5484] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb reported cstate change: term changed from 1 to 2, leader changed from b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) to c4e97cd51e9949339f937b6be4cefebb (127.6.5.5). New cstate: current_term: 2 leader_uuid: "c4e97cd51e9949339f937b6be4cefebb" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } } }
I20240502 08:17:11.065843 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.077320 9775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.095022 8165 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:11.095844 8295 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:11.115567 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:11.115718 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:11.135428 9645 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.292685 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.318127 9775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.325408 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:11.345803 9645 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.349578 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:11.355865 8295 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:11.359269 8165 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:11.391029 8185 raft_consensus.cc:1261] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Refusing update from remote peer b58bd494983a4a2885cda3387b1be902: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:17:11.392501 10844 consensus_queue.cc:1035] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:11.436036 8185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f87d008812d847bb89dcf4034387bb51"
dest_uuid: "b90db359bc744f2b82b69d97627fa7b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60286
I20240502 08:17:11.436216 8185 raft_consensus.cc:595] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:11.492797 8185 raft_consensus.cc:1261] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Refusing update from remote peer c4e97cd51e9949339f937b6be4cefebb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:17:11.493806 10862 consensus_queue.cc:1035] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:11.525384 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.538239 9775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.539938 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:11.547061 8295 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:11.563536 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:11.573720 8165 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:11.598204 9645 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.728267 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.751307 9775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.761596 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:11.767184 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:11.781352 8295 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:11.785507 8165 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:11.803555 9645 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.862108 8314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1fabef91cc4d4df3abfe3bdb393790e8"
dest_uuid: "89ae15f3455446339957f9559f3857da"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52090
I20240502 08:17:11.862309 8314 raft_consensus.cc:595] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:11.951531 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.963666 9775 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:11.980530 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:11.984411 8295 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:11.994575 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:11.997959 8165 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:12.027663 9645 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:12.059398 11040 raft_consensus.cc:979] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9: : Instructing follower 646203131e6c42a2a2e631374efbe5ee to start an election
I20240502 08:17:12.059629 11039 raft_consensus.cc:1067] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Signalling peer 646203131e6c42a2a2e631374efbe5ee to start an election
I20240502 08:17:12.060264 9795 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f87d008812d847bb89dcf4034387bb51"
dest_uuid: "646203131e6c42a2a2e631374efbe5ee"
from {username='slave'} at 127.6.5.1:50009
I20240502 08:17:12.060508 9795 raft_consensus.cc:482] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:12.060628 9795 raft_consensus.cc:3037] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:12.064029 9795 raft_consensus.cc:504] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } }
I20240502 08:17:12.065896 8185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f87d008812d847bb89dcf4034387bb51" candidate_uuid: "646203131e6c42a2a2e631374efbe5ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b90db359bc744f2b82b69d97627fa7b9"
I20240502 08:17:12.066102 8185 raft_consensus.cc:3032] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:12.066176 8185 raft_consensus.cc:729] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:17:12.066428 8185 consensus_queue.cc:260] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } }
I20240502 08:17:12.066663 8185 raft_consensus.cc:3037] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:12.069420 8185 raft_consensus.cc:2454] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 646203131e6c42a2a2e631374efbe5ee in term 2.
I20240502 08:17:12.069883 9731 leader_election.cc:302] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 646203131e6c42a2a2e631374efbe5ee, b90db359bc744f2b82b69d97627fa7b9; no voters:
I20240502 08:17:12.070199 10899 raft_consensus.cc:2781] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:12.071749 9795 leader_election.cc:288] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [CANDIDATE]: Term 2 election: Requested vote from peers b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:17:12.076939 10899 raft_consensus.cc:686] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [term 2 LEADER]: Becoming Leader. State: Replica: 646203131e6c42a2a2e631374efbe5ee, State: Running, Role: LEADER
I20240502 08:17:12.077644 10899 consensus_queue.cc:237] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } }
I20240502 08:17:12.080168 8044 catalog_manager.cc:5484] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee reported cstate change: term changed from 1 to 2, leader changed from b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) to 646203131e6c42a2a2e631374efbe5ee (127.6.5.6). New cstate: current_term: 2 leader_uuid: "646203131e6c42a2a2e631374efbe5ee" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } } }
I20240502 08:17:12.184187 9373 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:12.198781 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.208470 8295 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:12.210281 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.210855 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.225852 9645 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:12.231633 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:12.290897 11086 raft_consensus.cc:979] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da: : Instructing follower 00b1227597554ea18f7e125de6d2760e to start an election
I20240502 08:17:12.291157 11085 raft_consensus.cc:1067] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Signalling peer 00b1227597554ea18f7e125de6d2760e to start an election
I20240502 08:17:12.291923 9393 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1fabef91cc4d4df3abfe3bdb393790e8"
dest_uuid: "00b1227597554ea18f7e125de6d2760e"
from {username='slave'} at 127.6.5.2:33675
I20240502 08:17:12.292186 9393 raft_consensus.cc:482] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:12.292320 9393 raft_consensus.cc:3037] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:12.295630 9393 raft_consensus.cc:504] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } }
I20240502 08:17:12.303427 9393 leader_election.cc:288] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [CANDIDATE]: Term 2 election: Requested vote from peers 89ae15f3455446339957f9559f3857da (127.6.5.2:44539)
I20240502 08:17:12.303939 8314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fabef91cc4d4df3abfe3bdb393790e8" candidate_uuid: "00b1227597554ea18f7e125de6d2760e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "89ae15f3455446339957f9559f3857da"
I20240502 08:17:12.304167 8314 raft_consensus.cc:3032] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:12.304244 8314 raft_consensus.cc:729] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 89ae15f3455446339957f9559f3857da, State: Running, Role: LEADER
I20240502 08:17:12.304541 8314 consensus_queue.cc:260] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } }
I20240502 08:17:12.304759 8314 raft_consensus.cc:3037] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:12.307935 8314 raft_consensus.cc:2454] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b1227597554ea18f7e125de6d2760e in term 2.
I20240502 08:17:12.308585 9329 leader_election.cc:302] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 00b1227597554ea18f7e125de6d2760e, 89ae15f3455446339957f9559f3857da; no voters:
I20240502 08:17:12.310896 11037 raft_consensus.cc:2781] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:12.314931 11037 raft_consensus.cc:686] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [term 2 LEADER]: Becoming Leader. State: Replica: 00b1227597554ea18f7e125de6d2760e, State: Running, Role: LEADER
I20240502 08:17:12.315227 11037 consensus_queue.cc:237] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } }
I20240502 08:17:12.320799 8044 catalog_manager.cc:5484] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e reported cstate change: term changed from 1 to 2, leader changed from 89ae15f3455446339957f9559f3857da (127.6.5.2) to 00b1227597554ea18f7e125de6d2760e (127.6.5.3). New cstate: current_term: 2 leader_uuid: "00b1227597554ea18f7e125de6d2760e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } } }
I20240502 08:17:12.344094 8314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "74a88bd56f86453280555608afcb98f3"
dest_uuid: "89ae15f3455446339957f9559f3857da"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52124
I20240502 08:17:12.344273 8314 raft_consensus.cc:595] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:12.439646 8295 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:12.441998 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.442987 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.464038 8185 raft_consensus.cc:1261] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Refusing update from remote peer 646203131e6c42a2a2e631374efbe5ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:17:12.465754 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:12.466909 10899 consensus_queue.cc:1035] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:12.468936 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.470928 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.482028 9645 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:12.852494 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.854681 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.859087 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.865587 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:12.871266 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:12.884112 8295 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:12.896956 8314 raft_consensus.cc:1261] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Refusing update from remote peer 00b1227597554ea18f7e125de6d2760e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:17:12.897512 11037 consensus_queue.cc:1035] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [LEADER]: Connected to new peer: Peer: permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:12.916936 9645 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:12.925571 11168 raft_consensus.cc:979] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da: : Instructing follower c4e97cd51e9949339f937b6be4cefebb to start an election
I20240502 08:17:12.925846 11168 raft_consensus.cc:1067] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Signalling peer c4e97cd51e9949339f937b6be4cefebb to start an election
I20240502 08:17:12.926708 9665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74a88bd56f86453280555608afcb98f3"
dest_uuid: "c4e97cd51e9949339f937b6be4cefebb"
from {username='slave'} at 127.6.5.2:33999
I20240502 08:17:12.926945 9665 raft_consensus.cc:482] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:12.927045 9665 raft_consensus.cc:3037] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:12.929752 9665 raft_consensus.cc:504] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } }
I20240502 08:17:12.932410 8315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a88bd56f86453280555608afcb98f3" candidate_uuid: "c4e97cd51e9949339f937b6be4cefebb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "89ae15f3455446339957f9559f3857da"
I20240502 08:17:12.932674 8315 raft_consensus.cc:3032] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:12.932760 8315 raft_consensus.cc:729] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 89ae15f3455446339957f9559f3857da, State: Running, Role: LEADER
I20240502 08:17:12.933048 8315 consensus_queue.cc:260] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } }
I20240502 08:17:12.933279 8315 raft_consensus.cc:3037] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:12.936488 8315 raft_consensus.cc:2454] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4e97cd51e9949339f937b6be4cefebb in term 2.
I20240502 08:17:12.936677 9665 leader_election.cc:288] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [CANDIDATE]: Term 2 election: Requested vote from peers 89ae15f3455446339957f9559f3857da (127.6.5.2:44539)
I20240502 08:17:12.937486 9601 leader_election.cc:302] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 89ae15f3455446339957f9559f3857da, c4e97cd51e9949339f937b6be4cefebb; no voters:
I20240502 08:17:12.937917 11041 raft_consensus.cc:2781] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:12.954713 11041 raft_consensus.cc:686] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [term 2 LEADER]: Becoming Leader. State: Replica: c4e97cd51e9949339f937b6be4cefebb, State: Running, Role: LEADER
I20240502 08:17:12.955080 11041 consensus_queue.cc:237] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } }
I20240502 08:17:12.957831 8041 catalog_manager.cc:5484] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb reported cstate change: term changed from 1 to 2, leader changed from 89ae15f3455446339957f9559f3857da (127.6.5.2) to c4e97cd51e9949339f937b6be4cefebb (127.6.5.5). New cstate: current_term: 2 leader_uuid: "c4e97cd51e9949339f937b6be4cefebb" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } } }
I20240502 08:17:13.014953 9393 consensus_queue.cc:237] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } }
I20240502 08:17:13.016639 11175 raft_consensus.cc:2932] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } } }
I20240502 08:17:13.020722 8039 catalog_manager.cc:5484] T 0b3e2b0c939e4182bb851b4cdc802301 P 00b1227597554ea18f7e125de6d2760e reported cstate change: config changed from index 4 to 6, VOTER b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) evicted. New cstate: current_term: 2 leader_uuid: "00b1227597554ea18f7e125de6d2760e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } } }
I20240502 08:17:13.025102 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3e2b0c939e4182bb851b4cdc802301 with delete_type TABLET_DATA_TOMBSTONED (TS b90db359bc744f2b82b69d97627fa7b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46030
I20240502 08:17:13.025925 11199 tablet_replica.cc:330] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9: stopping tablet replica
I20240502 08:17:13.026363 11199 raft_consensus.cc:2229] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:13.026687 11199 raft_consensus.cc:2258] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:13.028398 11199 ts_tablet_manager.cc:1875] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:13.034096 11199 ts_tablet_manager.cc:1888] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:17:13.034240 11199 log.cc:1196] T 0b3e2b0c939e4182bb851b4cdc802301 P b90db359bc744f2b82b69d97627fa7b9: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/0b3e2b0c939e4182bb851b4cdc802301
I20240502 08:17:13.035223 8031 catalog_manager.cc:4830] TS b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591): tablet 0b3e2b0c939e4182bb851b4cdc802301 (table rebalance_test_table_2 [id=ec0e8263e141487d845dc940a90656fe]) successfully deleted
I20240502 08:17:13.112630 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.124538 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.133366 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.137806 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:13.157414 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:13.158016 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.166626 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.251034 9795 consensus_queue.cc:237] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } }
I20240502 08:17:13.252189 11042 raft_consensus.cc:2932] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 89ae15f3455446339957f9559f3857da (127.6.5.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } } }
I20240502 08:17:13.256819 8038 catalog_manager.cc:5484] T 3d96e8d4b891410980ae6a696799a60b P 646203131e6c42a2a2e631374efbe5ee reported cstate change: config changed from index 4 to 6, VOTER 89ae15f3455446339957f9559f3857da (127.6.5.2) evicted. New cstate: current_term: 2 leader_uuid: "646203131e6c42a2a2e631374efbe5ee" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } } }
I20240502 08:17:13.260402 8295 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d96e8d4b891410980ae6a696799a60b with delete_type TABLET_DATA_TOMBSTONED (TS 89ae15f3455446339957f9559f3857da not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41504
I20240502 08:17:13.261831 11257 tablet_replica.cc:330] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da: stopping tablet replica
I20240502 08:17:13.262274 11257 raft_consensus.cc:2229] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:13.262573 11257 raft_consensus.cc:2258] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:13.263839 11257 ts_tablet_manager.cc:1875] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:13.270121 11257 ts_tablet_manager.cc:1888] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:17:13.270259 11257 log.cc:1196] T 3d96e8d4b891410980ae6a696799a60b P 89ae15f3455446339957f9559f3857da: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/3d96e8d4b891410980ae6a696799a60b
I20240502 08:17:13.271117 8031 catalog_manager.cc:4830] TS 89ae15f3455446339957f9559f3857da (127.6.5.2:44539): tablet 3d96e8d4b891410980ae6a696799a60b (table rebalance_test_table_3 [id=d999095ea3574127ad81d93a2c85019d]) successfully deleted
I20240502 08:17:13.352725 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.359004 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.365425 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:13.372979 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.385273 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:13.406107 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.416501 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.500233 9925 consensus_queue.cc:237] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } }
I20240502 08:17:13.501423 10668 raft_consensus.cc:2932] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } } }
I20240502 08:17:13.505676 8040 catalog_manager.cc:5484] T afc5bc8efa7d4a3fb8970bf100c754a2 P 6e2d5eaefade459c8e1fce0db41f2aca reported cstate change: config changed from index 4 to 6, VOTER b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) evicted. New cstate: current_term: 2 leader_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } } }
I20240502 08:17:13.509325 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet afc5bc8efa7d4a3fb8970bf100c754a2 with delete_type TABLET_DATA_TOMBSTONED (TS b90db359bc744f2b82b69d97627fa7b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46030
I20240502 08:17:13.509677 11199 tablet_replica.cc:330] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9: stopping tablet replica
I20240502 08:17:13.509894 11199 raft_consensus.cc:2229] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:13.510128 11199 raft_consensus.cc:2258] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:13.510845 11199 ts_tablet_manager.cc:1875] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:13.515637 11199 ts_tablet_manager.cc:1888] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:17:13.515861 11199 log.cc:1196] T afc5bc8efa7d4a3fb8970bf100c754a2 P b90db359bc744f2b82b69d97627fa7b9: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/afc5bc8efa7d4a3fb8970bf100c754a2
I20240502 08:17:13.516644 8031 catalog_manager.cc:4830] TS b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591): tablet afc5bc8efa7d4a3fb8970bf100c754a2 (table rebalance_test_table_4 [id=bf5ea02dca1c4ab89d3224c8e4a45df1]) successfully deleted
I20240502 08:17:13.524596 8314 raft_consensus.cc:1261] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Refusing update from remote peer c4e97cd51e9949339f937b6be4cefebb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:17:13.525122 11041 consensus_queue.cc:1035] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:13.602947 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.603006 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.620926 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.620970 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:13.622321 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.675724 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:13.677098 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.754159 9925 consensus_queue.cc:237] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } }
I20240502 08:17:13.755411 10668 raft_consensus.cc:2932] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 89ae15f3455446339957f9559f3857da (127.6.5.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } } }
I20240502 08:17:13.758931 8038 catalog_manager.cc:5484] T aa41e5f5e3ef4dec89a0a39198f7a53c P 6e2d5eaefade459c8e1fce0db41f2aca reported cstate change: config changed from index 4 to 6, VOTER 89ae15f3455446339957f9559f3857da (127.6.5.2) evicted. New cstate: current_term: 2 leader_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6e2d5eaefade459c8e1fce0db41f2aca" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 39789 } } }
I20240502 08:17:13.762249 8295 tablet_service.cc:1514] Processing DeleteTablet for tablet aa41e5f5e3ef4dec89a0a39198f7a53c with delete_type TABLET_DATA_TOMBSTONED (TS 89ae15f3455446339957f9559f3857da not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41504
I20240502 08:17:13.762638 11257 tablet_replica.cc:330] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da: stopping tablet replica
I20240502 08:17:13.762831 11257 raft_consensus.cc:2229] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:13.763015 11257 raft_consensus.cc:2258] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:13.763581 11257 ts_tablet_manager.cc:1875] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:13.768216 11257 ts_tablet_manager.cc:1888] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:17:13.768325 11257 log.cc:1196] T aa41e5f5e3ef4dec89a0a39198f7a53c P 89ae15f3455446339957f9559f3857da: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa41e5f5e3ef4dec89a0a39198f7a53c
I20240502 08:17:13.768939 8031 catalog_manager.cc:4830] TS 89ae15f3455446339957f9559f3857da (127.6.5.2:44539): tablet aa41e5f5e3ef4dec89a0a39198f7a53c (table rebalance_test_table_3 [id=d999095ea3574127ad81d93a2c85019d]) successfully deleted
I20240502 08:17:13.856009 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.856904 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.871261 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.872502 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.884613 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:13.893715 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:13.930785 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:13.999500 9665 consensus_queue.cc:237] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } }
I20240502 08:17:14.000662 11320 raft_consensus.cc:2932] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } } }
I20240502 08:17:14.004732 8043 catalog_manager.cc:5484] T 72b8cb6631624a839ec2b3a77abaa1df P c4e97cd51e9949339f937b6be4cefebb reported cstate change: config changed from index 4 to 6, VOTER b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) evicted. New cstate: current_term: 2 leader_uuid: "c4e97cd51e9949339f937b6be4cefebb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } } }
I20240502 08:17:14.008426 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet 72b8cb6631624a839ec2b3a77abaa1df with delete_type TABLET_DATA_TOMBSTONED (TS b90db359bc744f2b82b69d97627fa7b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46030
I20240502 08:17:14.008816 11199 tablet_replica.cc:330] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9: stopping tablet replica
I20240502 08:17:14.009027 11199 raft_consensus.cc:2229] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:14.009225 11199 raft_consensus.cc:2258] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:14.009830 11199 ts_tablet_manager.cc:1875] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:14.015322 11199 ts_tablet_manager.cc:1888] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:17:14.015431 11199 log.cc:1196] T 72b8cb6631624a839ec2b3a77abaa1df P b90db359bc744f2b82b69d97627fa7b9: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/72b8cb6631624a839ec2b3a77abaa1df
I20240502 08:17:14.016081 8031 catalog_manager.cc:4830] TS b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591): tablet 72b8cb6631624a839ec2b3a77abaa1df (table rebalance_test_table_2 [id=ec0e8263e141487d845dc940a90656fe]) successfully deleted
I20240502 08:17:14.105612 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.115793 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.123106 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.125722 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.128880 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:14.134865 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:14.147223 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.234380 9526 consensus_queue.cc:237] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } }
I20240502 08:17:14.235534 11400 raft_consensus.cc:2932] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } } }
I20240502 08:17:14.239665 8039 catalog_manager.cc:5484] T bdce98c845724dc3a9cedf89e3bec62c P b58bd494983a4a2885cda3387b1be902 reported cstate change: config changed from index 4 to 6, VOTER b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) evicted. New cstate: current_term: 2 leader_uuid: "b58bd494983a4a2885cda3387b1be902" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b58bd494983a4a2885cda3387b1be902" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 35529 } } }
I20240502 08:17:14.243321 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet bdce98c845724dc3a9cedf89e3bec62c with delete_type TABLET_DATA_TOMBSTONED (TS b90db359bc744f2b82b69d97627fa7b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46030
I20240502 08:17:14.243686 11199 tablet_replica.cc:330] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9: stopping tablet replica
I20240502 08:17:14.243898 11199 raft_consensus.cc:2229] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:14.244117 11199 raft_consensus.cc:2258] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:14.244871 11199 ts_tablet_manager.cc:1875] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:14.251124 11199 ts_tablet_manager.cc:1888] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:17:14.251259 11199 log.cc:1196] T bdce98c845724dc3a9cedf89e3bec62c P b90db359bc744f2b82b69d97627fa7b9: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/bdce98c845724dc3a9cedf89e3bec62c
I20240502 08:17:14.251960 8031 catalog_manager.cc:4830] TS b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591): tablet bdce98c845724dc3a9cedf89e3bec62c (table rebalance_test_table_0 [id=bfa22e199f7d4cd1aa040b3473937ac8]) successfully deleted
I20240502 08:17:14.352331 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.352331 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.359525 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.364953 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.365423 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:14.378444 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.379112 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:14.472112 9795 consensus_queue.cc:237] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } }
I20240502 08:17:14.473320 11453 raft_consensus.cc:2932] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } } }
I20240502 08:17:14.477514 8039 catalog_manager.cc:5484] T f87d008812d847bb89dcf4034387bb51 P 646203131e6c42a2a2e631374efbe5ee reported cstate change: config changed from index 4 to 6, VOTER b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) evicted. New cstate: current_term: 2 leader_uuid: "646203131e6c42a2a2e631374efbe5ee" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "646203131e6c42a2a2e631374efbe5ee" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 34035 } } }
I20240502 08:17:14.480943 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet f87d008812d847bb89dcf4034387bb51 with delete_type TABLET_DATA_TOMBSTONED (TS b90db359bc744f2b82b69d97627fa7b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46030
I20240502 08:17:14.481232 11199 tablet_replica.cc:330] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9: stopping tablet replica
I20240502 08:17:14.481441 11199 raft_consensus.cc:2229] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:14.481635 11199 raft_consensus.cc:2258] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:14.482254 11199 ts_tablet_manager.cc:1875] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:14.487514 11199 ts_tablet_manager.cc:1888] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:17:14.487622 11199 log.cc:1196] T f87d008812d847bb89dcf4034387bb51 P b90db359bc744f2b82b69d97627fa7b9: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/f87d008812d847bb89dcf4034387bb51
I20240502 08:17:14.488248 8031 catalog_manager.cc:4830] TS b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591): tablet f87d008812d847bb89dcf4034387bb51 (table rebalance_test_table_1 [id=37605e3d381045239bd078121ab54c76]) successfully deleted
I20240502 08:17:14.567541 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.589510 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.590679 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.607839 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.607904 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:14.627537 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:14.632167 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.711223 9393 consensus_queue.cc:237] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } }
I20240502 08:17:14.712467 11564 raft_consensus.cc:2932] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 89ae15f3455446339957f9559f3857da (127.6.5.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } } }
I20240502 08:17:14.716684 8040 catalog_manager.cc:5484] T 1fabef91cc4d4df3abfe3bdb393790e8 P 00b1227597554ea18f7e125de6d2760e reported cstate change: config changed from index 4 to 6, VOTER 89ae15f3455446339957f9559f3857da (127.6.5.2) evicted. New cstate: current_term: 2 leader_uuid: "00b1227597554ea18f7e125de6d2760e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "00b1227597554ea18f7e125de6d2760e" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 43567 } } }
I20240502 08:17:14.720297 8295 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fabef91cc4d4df3abfe3bdb393790e8 with delete_type TABLET_DATA_TOMBSTONED (TS 89ae15f3455446339957f9559f3857da not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41504
I20240502 08:17:14.721524 11588 tablet_replica.cc:330] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da: stopping tablet replica
I20240502 08:17:14.721724 11588 raft_consensus.cc:2229] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:14.722697 11588 raft_consensus.cc:2258] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:14.723798 11588 ts_tablet_manager.cc:1875] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:14.732457 11588 ts_tablet_manager.cc:1888] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:17:14.732645 11588 log.cc:1196] T 1fabef91cc4d4df3abfe3bdb393790e8 P 89ae15f3455446339957f9559f3857da: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/1fabef91cc4d4df3abfe3bdb393790e8
I20240502 08:17:14.733373 8031 catalog_manager.cc:4830] TS 89ae15f3455446339957f9559f3857da (127.6.5.2:44539): tablet 1fabef91cc4d4df3abfe3bdb393790e8 (table rebalance_test_table_1 [id=37605e3d381045239bd078121ab54c76]) successfully deleted
I20240502 08:17:14.815215 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.815327 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.831936 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.838414 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:14.849587 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:14.867259 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.882966 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:14.958535 9665 consensus_queue.cc:237] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } }
I20240502 08:17:14.960156 11565 raft_consensus.cc:2932] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 89ae15f3455446339957f9559f3857da (127.6.5.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } } }
I20240502 08:17:14.964605 8040 catalog_manager.cc:5484] T 74a88bd56f86453280555608afcb98f3 P c4e97cd51e9949339f937b6be4cefebb reported cstate change: config changed from index 4 to 6, VOTER 89ae15f3455446339957f9559f3857da (127.6.5.2) evicted. New cstate: current_term: 2 leader_uuid: "c4e97cd51e9949339f937b6be4cefebb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c4e97cd51e9949339f937b6be4cefebb" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 34875 } } }
I20240502 08:17:14.967854 8295 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a88bd56f86453280555608afcb98f3 with delete_type TABLET_DATA_TOMBSTONED (TS 89ae15f3455446339957f9559f3857da not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41504
I20240502 08:17:14.968178 11588 tablet_replica.cc:330] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da: stopping tablet replica
I20240502 08:17:14.968356 11588 raft_consensus.cc:2229] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:14.968526 11588 raft_consensus.cc:2258] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:14.969125 11588 ts_tablet_manager.cc:1875] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:14.973227 11588 ts_tablet_manager.cc:1888] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:17:14.973320 11588 log.cc:1196] T 74a88bd56f86453280555608afcb98f3 P 89ae15f3455446339957f9559f3857da: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/74a88bd56f86453280555608afcb98f3
I20240502 08:17:14.973942 8031 catalog_manager.cc:4830] TS 89ae15f3455446339957f9559f3857da (127.6.5.2:44539): tablet 74a88bd56f86453280555608afcb98f3 (table rebalance_test_table_4 [id=bf5ea02dca1c4ab89d3224c8e4a45df1]) successfully deleted
I20240502 08:17:15.338649 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:15.369400 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:15.379773 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:15.382006 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:15.387789 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:15.394667 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:15.403916 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:15.559000 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:15.576586 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:15.577288 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:15.578109 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:15.594944 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:15.599462 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:15.601452 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.285549 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.306962 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.312767 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.316605 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:16.361135 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.362027 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:16.368942 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.540518 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.558641 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.564666 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.573082 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:16.574879 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.593710 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:16.600714 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.863147 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.872408 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.874938 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.887295 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:16.890105 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:16.900789 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:16.921339 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+------------------+---------
1e7a91bc1fb549619d016b96f81b6ff5 | 127.6.5.62:44177 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+------------------+-------------------------
builtin_ntp_servers | 127.6.5.20:43353 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-----------------+---------+----------+----------------+-----------------
00b1227597554ea18f7e125de6d2760e | 127.6.5.3:43567 | HEALTHY | <none> | 2 | 0
646203131e6c42a2a2e631374efbe5ee | 127.6.5.6:34035 | HEALTHY | <none> | 2 | 0
6e2d5eaefade459c8e1fce0db41f2aca | 127.6.5.7:39789 | HEALTHY | <none> | 2 | 0
89ae15f3455446339957f9559f3857da | 127.6.5.2:44539 | HEALTHY | <none> | 3 | 0
b58bd494983a4a2885cda3387b1be902 | 127.6.5.4:35529 | HEALTHY | <none> | 2 | 0
b90db359bc744f2b82b69d97627fa7b9 | 127.6.5.1:35591 | HEALTHY | <none> | 3 | 0
c4e97cd51e9949339f937b6be4cefebb | 127.6.5.5:34875 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.6.5.1 | experimental | 127.6.5.1:35591
local_ip_for_outbound_sockets | 127.6.5.2 | experimental | 127.6.5.2:44539
local_ip_for_outbound_sockets | 127.6.5.3 | experimental | 127.6.5.3:43567
local_ip_for_outbound_sockets | 127.6.5.4 | experimental | 127.6.5.4:35529
local_ip_for_outbound_sockets | 127.6.5.5 | experimental | 127.6.5.5:34875
local_ip_for_outbound_sockets | 127.6.5.6 | experimental | 127.6.5.6:34035
local_ip_for_outbound_sockets | 127.6.5.7 | experimental | 127.6.5.7:39789
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.6.5.1:35591
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.6.5.2:44539
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.6.5.3:43567
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.6.5.4:35529
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.6.5.5:34875
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.6.5.6:34035
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.6.5.7:39789
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+------------------+-------------------------
builtin_ntp_servers | 127.6.5.20:43353 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 08:17:16.976806 6164 log_verifier.cc:126] Checking tablet 0b3e2b0c939e4182bb851b4cdc802301
I20240502 08:17:16.992410 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0b3e2b0c939e4182bb851b4cdc802301
I20240502 08:17:16.992573 6164 log_verifier.cc:126] Checking tablet 1fabef91cc4d4df3abfe3bdb393790e8
I20240502 08:17:17.004743 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 1fabef91cc4d4df3abfe3bdb393790e8
I20240502 08:17:17.004863 6164 log_verifier.cc:126] Checking tablet 3d96e8d4b891410980ae6a696799a60b
I20240502 08:17:17.017817 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3d96e8d4b891410980ae6a696799a60b
I20240502 08:17:17.017954 6164 log_verifier.cc:126] Checking tablet 4a905a7206774ff096194077781dcd45
I20240502 08:17:17.032012 6164 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4a905a7206774ff096194077781dcd45
I20240502 08:17:17.032150 6164 log_verifier.cc:126] Checking tablet 72b8cb6631624a839ec2b3a77abaa1df
I20240502 08:17:17.044159 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 72b8cb6631624a839ec2b3a77abaa1df
I20240502 08:17:17.044291 6164 log_verifier.cc:126] Checking tablet 74a88bd56f86453280555608afcb98f3
I20240502 08:17:17.056592 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 74a88bd56f86453280555608afcb98f3
I20240502 08:17:17.056716 6164 log_verifier.cc:126] Checking tablet 9802b878c2f6463983a4029b7031004d
I20240502 08:17:17.068222 6164 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9802b878c2f6463983a4029b7031004d
I20240502 08:17:17.068339 6164 log_verifier.cc:126] Checking tablet aa41e5f5e3ef4dec89a0a39198f7a53c
I20240502 08:17:17.080421 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet aa41e5f5e3ef4dec89a0a39198f7a53c
I20240502 08:17:17.080544 6164 log_verifier.cc:126] Checking tablet afc5bc8efa7d4a3fb8970bf100c754a2
I20240502 08:17:17.092649 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet afc5bc8efa7d4a3fb8970bf100c754a2
I20240502 08:17:17.092775 6164 log_verifier.cc:126] Checking tablet bdce98c845724dc3a9cedf89e3bec62c
I20240502 08:17:17.105019 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet bdce98c845724dc3a9cedf89e3bec62c
I20240502 08:17:17.105141 6164 log_verifier.cc:126] Checking tablet c772b9342a314c3191a5caaf4395c94f
I20240502 08:17:17.116468 6164 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c772b9342a314c3191a5caaf4395c94f
I20240502 08:17:17.116580 6164 log_verifier.cc:126] Checking tablet d9a643e4d8dc4eeba0a31fc105c141a0
I20240502 08:17:17.128059 6164 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d9a643e4d8dc4eeba0a31fc105c141a0
I20240502 08:17:17.128168 6164 log_verifier.cc:126] Checking tablet da122388bec1424bb9e07919dd2d3cf0
I20240502 08:17:17.135432 6164 log_verifier.cc:177] Verified matching terms for 2 ops in tablet da122388bec1424bb9e07919dd2d3cf0
I20240502 08:17:17.135551 6164 log_verifier.cc:126] Checking tablet dba45a53e0ea4de1b03d12e36a8bb7bb
I20240502 08:17:17.146908 6164 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dba45a53e0ea4de1b03d12e36a8bb7bb
I20240502 08:17:17.147019 6164 log_verifier.cc:126] Checking tablet f766891d6cea4e1e827d3233c2e17a27
I20240502 08:17:17.158428 6164 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f766891d6cea4e1e827d3233c2e17a27
I20240502 08:17:17.158560 6164 log_verifier.cc:126] Checking tablet f87d008812d847bb89dcf4034387bb51
I20240502 08:17:17.170352 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f87d008812d847bb89dcf4034387bb51
I20240502 08:17:17.173271 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.8:0
--local_ip_for_outbound_sockets=127.6.5.8
--webserver_interface=127.6.5.8
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--builtin_ntp_servers=127.6.5.20:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:17:17.502564 11863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:17.502820 11863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:17.502900 11863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:17.510509 11863 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:17.510612 11863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:17.510726 11863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.8
I20240502 08:17:17.519474 11863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.5.8
--webserver_port=0
--tserver_master_addrs=127.6.5.62:44177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.8
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:17.520596 11863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:17.522100 11863 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:17.533874 11871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:17.534309 11868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:17.534723 11869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:17.535250 11863 server_base.cc:1034] running on GCE node
I20240502 08:17:17.535743 11863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:17.536530 11863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:17.537705 11863 hybrid_clock.cc:648] HybridClock initialized: now 1714637837537651 us; error 46 us; skew 500 ppm
I20240502 08:17:17.537987 11863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:17.540551 11863 webserver.cc:460] Webserver started at http://127.6.5.8:37647/ using document root <none> and password file <none>
I20240502 08:17:17.541173 11863 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:17.541258 11863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:17.541503 11863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:17.543833 11863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "c7e49faad12d44b79209c8df4a6e61af"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-h7h6"
I20240502 08:17:17.544510 11863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "c7e49faad12d44b79209c8df4a6e61af"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-h7h6"
I20240502 08:17:17.549111 11863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:17:17.552120 11877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:17.553354 11863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:17.553483 11863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c7e49faad12d44b79209c8df4a6e61af"
format_stamp: "Formatted at 2024-05-02 08:17:17 on dist-test-slave-h7h6"
I20240502 08:17:17.553651 11863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:17.572717 11863 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:17.573361 11863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:17.573587 11863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:17.574404 11863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:17.575644 11863 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:17.575723 11863 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:17.575817 11863 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:17.575891 11863 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:17.606259 11863 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.8:33883
I20240502 08:17:17.606323 11990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.8:33883 every 8 connection(s)
I20240502 08:17:17.607417 11863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:17:17.609316 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 11863
I20240502 08:17:17.609537 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 08:17:17.621369 11991 heartbeater.cc:347] Connected to a master server at 127.6.5.62:44177
I20240502 08:17:17.621621 11991 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:17.622246 11991 heartbeater.cc:511] Master 127.6.5.62:44177 requested a full tablet report, sending...
I20240502 08:17:17.623613 8041 ts_manager.cc:194] Registered new tserver with Master: c7e49faad12d44b79209c8df4a6e61af (127.6.5.8:33883)
I20240502 08:17:17.624696 8041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.8:51815
I20240502 08:17:18.122083 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:18.122083 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:18.125597 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:18.137378 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:18.140733 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:18.145429 11925 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:18.174144 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:18.178673 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:18.361685 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:18.368050 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:18.381412 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:18.385293 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:18.387965 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:18.388949 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:18.389585 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:18.418728 11925 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:18.509102 8184 consensus_queue.cc:237] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } }
W20240502 08:17:18.517777 8118 consensus_peers.cc:485] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 -> Peer c7e49faad12d44b79209c8df4a6e61af (127.6.5.8:33883): Couldn't send request to peer c7e49faad12d44b79209c8df4a6e61af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9802b878c2f6463983a4029b7031004d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:18.626849 11991 heartbeater.cc:503] Master 127.6.5.62:44177 was elected leader, sending a full tablet report...
I20240502 08:17:18.981564 12098 ts_tablet_manager.cc:902] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af: Initiating tablet copy from peer b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:17:18.983266 12098 tablet_copy_client.cc:308] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:35591
I20240502 08:17:18.989245 8205 tablet_copy_service.cc:140] P b90db359bc744f2b82b69d97627fa7b9: Received BeginTabletCopySession request for tablet 9802b878c2f6463983a4029b7031004d from peer c7e49faad12d44b79209c8df4a6e61af ({username='slave'} at 127.6.5.8:55553)
I20240502 08:17:18.989398 8205 tablet_copy_service.cc:161] P b90db359bc744f2b82b69d97627fa7b9: Beginning new tablet copy session on tablet 9802b878c2f6463983a4029b7031004d from peer c7e49faad12d44b79209c8df4a6e61af at {username='slave'} at 127.6.5.8:55553: session id = c7e49faad12d44b79209c8df4a6e61af-9802b878c2f6463983a4029b7031004d
I20240502 08:17:18.991408 8205 tablet_copy_source_session.cc:205] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:18.995944 12098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9802b878c2f6463983a4029b7031004d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:19.004738 12098 tablet_copy_client.cc:783] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:19.004971 12098 tablet_copy_client.cc:647] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:19.006877 12098 tablet_copy_client.cc:523] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:19.009819 12098 tablet_bootstrap.cc:492] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af: Bootstrap starting.
I20240502 08:17:19.019326 12098 log.cc:824] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:19.048238 12098 tablet_bootstrap.cc:492] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:19.048599 12098 tablet_bootstrap.cc:492] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af: Bootstrap complete.
I20240502 08:17:19.048914 12098 ts_tablet_manager.cc:1367] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af: Time spent bootstrapping tablet: real 0.039s user 0.028s sys 0.012s
I20240502 08:17:19.052866 12098 raft_consensus.cc:348] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } }
I20240502 08:17:19.053576 12098 raft_consensus.cc:729] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e49faad12d44b79209c8df4a6e61af, State: Initialized, Role: FOLLOWER
I20240502 08:17:19.053939 12098 consensus_queue.cc:260] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } }
I20240502 08:17:19.055205 12098 ts_tablet_manager.cc:1398] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20240502 08:17:19.056066 8205 tablet_copy_service.cc:342] P b90db359bc744f2b82b69d97627fa7b9: Request end of tablet copy session c7e49faad12d44b79209c8df4a6e61af-9802b878c2f6463983a4029b7031004d received from {username='slave'} at 127.6.5.8:55553
I20240502 08:17:19.056175 8205 tablet_copy_service.cc:433] P b90db359bc744f2b82b69d97627fa7b9: ending tablet copy session c7e49faad12d44b79209c8df4a6e61af-9802b878c2f6463983a4029b7031004d on tablet 9802b878c2f6463983a4029b7031004d with peer c7e49faad12d44b79209c8df4a6e61af
I20240502 08:17:19.537724 11945 raft_consensus.cc:1203] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:19.539265 12108 raft_consensus.cc:2932] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c7e49faad12d44b79209c8df4a6e61af (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } } }
I20240502 08:17:19.540083 11945 raft_consensus.cc:2932] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c7e49faad12d44b79209c8df4a6e61af (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } } }
I20240502 08:17:19.542290 12108 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:60502 (request call id 0) took 1033 ms. Trace:
I20240502 08:17:19.542503 12108 rpcz_store.cc:276] 0502 08:17:18.508552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:17:18.508614 (+ 62us) service_pool.cc:225] Handling call
0502 08:17:18.509812 (+ 1198us) log.cc:842] Serialized 248 byte log entry
0502 08:17:19.542243 (+1032431us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":56,"raft.queue_time_us":368,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":368,"spinlock_wait_cycles":236672,"thread_start_us":911,"threads_started":2,"wal-append.queue_time_us":777,"wal-append.run_cpu_time_us":609,"wal-append.run_wall_time_us":1000661}
I20240502 08:17:19.544502 8043 catalog_manager.cc:5484] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 reported cstate change: config changed from index -1 to 4, VOTER c7e49faad12d44b79209c8df4a6e61af (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "b90db359bc744f2b82b69d97627fa7b9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } } }
I20240502 08:17:19.578372 8314 consensus_queue.cc:237] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } }
W20240502 08:17:19.586239 8248 consensus_peers.cc:485] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da -> Peer c7e49faad12d44b79209c8df4a6e61af (127.6.5.8:33883): Couldn't send request to peer c7e49faad12d44b79209c8df4a6e61af. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c772b9342a314c3191a5caaf4395c94f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:19.966851 12136 ts_tablet_manager.cc:902] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af: Initiating tablet copy from peer 89ae15f3455446339957f9559f3857da (127.6.5.2:44539)
I20240502 08:17:19.968624 12136 tablet_copy_client.cc:308] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:44539
I20240502 08:17:19.974193 8335 tablet_copy_service.cc:140] P 89ae15f3455446339957f9559f3857da: Received BeginTabletCopySession request for tablet c772b9342a314c3191a5caaf4395c94f from peer c7e49faad12d44b79209c8df4a6e61af ({username='slave'} at 127.6.5.8:40025)
I20240502 08:17:19.974344 8335 tablet_copy_service.cc:161] P 89ae15f3455446339957f9559f3857da: Beginning new tablet copy session on tablet c772b9342a314c3191a5caaf4395c94f from peer c7e49faad12d44b79209c8df4a6e61af at {username='slave'} at 127.6.5.8:40025: session id = c7e49faad12d44b79209c8df4a6e61af-c772b9342a314c3191a5caaf4395c94f
I20240502 08:17:19.976290 8335 tablet_copy_source_session.cc:205] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:19.977509 12136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c772b9342a314c3191a5caaf4395c94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:19.982131 12136 tablet_copy_client.cc:783] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:19.982831 12136 tablet_copy_client.cc:647] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:19.984279 12136 tablet_copy_client.cc:523] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:19.986420 12136 tablet_bootstrap.cc:492] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af: Bootstrap starting.
I20240502 08:17:20.015300 12136 tablet_bootstrap.cc:492] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:20.015638 12136 tablet_bootstrap.cc:492] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af: Bootstrap complete.
I20240502 08:17:20.015904 12136 ts_tablet_manager.cc:1367] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.004s
I20240502 08:17:20.016749 12136 raft_consensus.cc:348] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } }
I20240502 08:17:20.017139 12136 raft_consensus.cc:729] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e49faad12d44b79209c8df4a6e61af, State: Initialized, Role: FOLLOWER
I20240502 08:17:20.017319 12136 consensus_queue.cc:260] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } }
I20240502 08:17:20.017941 12136 ts_tablet_manager.cc:1398] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:20.018846 8335 tablet_copy_service.cc:342] P 89ae15f3455446339957f9559f3857da: Request end of tablet copy session c7e49faad12d44b79209c8df4a6e61af-c772b9342a314c3191a5caaf4395c94f received from {username='slave'} at 127.6.5.8:40025
I20240502 08:17:20.019026 8335 tablet_copy_service.cc:433] P 89ae15f3455446339957f9559f3857da: ending tablet copy session c7e49faad12d44b79209c8df4a6e61af-c772b9342a314c3191a5caaf4395c94f on tablet c772b9342a314c3191a5caaf4395c94f with peer c7e49faad12d44b79209c8df4a6e61af
I20240502 08:17:20.368583 11944 raft_consensus.cc:1203] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:17:20.369940 12143 raft_consensus.cc:2932] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c7e49faad12d44b79209c8df4a6e61af (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } } }
I20240502 08:17:20.370698 11944 raft_consensus.cc:2932] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c7e49faad12d44b79209c8df4a6e61af (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } } }
I20240502 08:17:20.376793 8043 catalog_manager.cc:5484] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af reported cstate change: config changed from index -1 to 4, VOTER c7e49faad12d44b79209c8df4a6e61af (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "89ae15f3455446339957f9559f3857da" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } } }
I20240502 08:17:20.489264 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.501538 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:20.509011 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.514118 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.515436 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.542259 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:20.557685 11925 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:20.588337 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.641109 8314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c772b9342a314c3191a5caaf4395c94f"
dest_uuid: "89ae15f3455446339957f9559f3857da"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59520
I20240502 08:17:20.641321 8314 raft_consensus.cc:595] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:20.724295 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.726148 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.731102 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.745779 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.746162 8295 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:20.768580 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:20.789398 12143 raft_consensus.cc:979] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da: : Instructing follower c7e49faad12d44b79209c8df4a6e61af to start an election
I20240502 08:17:20.789595 12143 raft_consensus.cc:1067] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Signalling peer c7e49faad12d44b79209c8df4a6e61af to start an election
I20240502 08:17:20.791069 11944 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c772b9342a314c3191a5caaf4395c94f"
dest_uuid: "c7e49faad12d44b79209c8df4a6e61af"
from {username='slave'} at 127.6.5.2:49431
I20240502 08:17:20.791363 11944 raft_consensus.cc:482] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:20.791565 11944 raft_consensus.cc:3037] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:20.791612 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.794835 11944 raft_consensus.cc:504] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } }
I20240502 08:17:20.796845 11944 leader_election.cc:288] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [CANDIDATE]: Term 2 election: Requested vote from peers 89ae15f3455446339957f9559f3857da (127.6.5.2:44539)
I20240502 08:17:20.796855 8314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c772b9342a314c3191a5caaf4395c94f" candidate_uuid: "c7e49faad12d44b79209c8df4a6e61af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "89ae15f3455446339957f9559f3857da"
I20240502 08:17:20.797089 8314 raft_consensus.cc:3032] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:20.797205 8314 raft_consensus.cc:729] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 89ae15f3455446339957f9559f3857da, State: Running, Role: LEADER
I20240502 08:17:20.797467 8314 consensus_queue.cc:260] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } }
I20240502 08:17:20.797786 8314 raft_consensus.cc:3037] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:20.800695 8314 raft_consensus.cc:2454] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e49faad12d44b79209c8df4a6e61af in term 2.
I20240502 08:17:20.801303 11881 leader_election.cc:302] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 89ae15f3455446339957f9559f3857da, c7e49faad12d44b79209c8df4a6e61af; no voters:
I20240502 08:17:20.801898 12106 raft_consensus.cc:2781] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:20.830580 12106 raft_consensus.cc:686] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [term 2 LEADER]: Becoming Leader. State: Replica: c7e49faad12d44b79209c8df4a6e61af, State: Running, Role: LEADER
I20240502 08:17:20.831188 12106 consensus_queue.cc:237] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } }
I20240502 08:17:20.835762 11925 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:20.836197 8039 catalog_manager.cc:5484] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af reported cstate change: term changed from 1 to 2, leader changed from 89ae15f3455446339957f9559f3857da (127.6.5.2) to c7e49faad12d44b79209c8df4a6e61af (127.6.5.8). New cstate: current_term: 2 leader_uuid: "c7e49faad12d44b79209c8df4a6e61af" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } } }
I20240502 08:17:20.973424 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.990425 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.997041 8295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:20.999050 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.017107 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.031579 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:21.050788 11925 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:21.058177 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.120184 8184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9802b878c2f6463983a4029b7031004d"
dest_uuid: "b90db359bc744f2b82b69d97627fa7b9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37444
I20240502 08:17:21.120409 8184 raft_consensus.cc:595] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:21.201535 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.216084 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.225488 8295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.229667 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.241748 8314 raft_consensus.cc:1261] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Refusing update from remote peer c7e49faad12d44b79209c8df4a6e61af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:17:21.242249 8165 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:21.242640 12106 consensus_queue.cc:1035] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [LEADER]: Connected to new peer: Peer: permanent_uuid: "89ae15f3455446339957f9559f3857da" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 44539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:21.267128 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.277143 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.278231 11925 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:21.397269 12193 raft_consensus.cc:979] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9: : Instructing follower c7e49faad12d44b79209c8df4a6e61af to start an election
I20240502 08:17:21.397421 12193 raft_consensus.cc:1067] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Signalling peer c7e49faad12d44b79209c8df4a6e61af to start an election
I20240502 08:17:21.398037 11944 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9802b878c2f6463983a4029b7031004d"
dest_uuid: "c7e49faad12d44b79209c8df4a6e61af"
from {username='slave'} at 127.6.5.1:48539
I20240502 08:17:21.398223 11944 raft_consensus.cc:482] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:21.398319 11944 raft_consensus.cc:3037] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:21.400183 11944 raft_consensus.cc:504] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } }
I20240502 08:17:21.400697 11944 leader_election.cc:288] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [CANDIDATE]: Term 2 election: Requested vote from peers b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591)
I20240502 08:17:21.401140 8184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9802b878c2f6463983a4029b7031004d" candidate_uuid: "c7e49faad12d44b79209c8df4a6e61af" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b90db359bc744f2b82b69d97627fa7b9"
I20240502 08:17:21.401320 8184 raft_consensus.cc:3032] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:21.401383 8184 raft_consensus.cc:729] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b90db359bc744f2b82b69d97627fa7b9, State: Running, Role: LEADER
I20240502 08:17:21.401576 8184 consensus_queue.cc:260] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } }
I20240502 08:17:21.401808 8184 raft_consensus.cc:3037] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:21.403678 8184 raft_consensus.cc:2454] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7e49faad12d44b79209c8df4a6e61af in term 2.
I20240502 08:17:21.404032 11881 leader_election.cc:302] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b90db359bc744f2b82b69d97627fa7b9, c7e49faad12d44b79209c8df4a6e61af; no voters:
I20240502 08:17:21.404294 12106 raft_consensus.cc:2781] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:21.404414 12106 raft_consensus.cc:686] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [term 2 LEADER]: Becoming Leader. State: Replica: c7e49faad12d44b79209c8df4a6e61af, State: Running, Role: LEADER
I20240502 08:17:21.404668 12106 consensus_queue.cc:237] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } }
I20240502 08:17:21.406805 8041 catalog_manager.cc:5484] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af reported cstate change: term changed from 1 to 2, leader changed from b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) to c7e49faad12d44b79209c8df4a6e61af (127.6.5.8). New cstate: current_term: 2 leader_uuid: "c7e49faad12d44b79209c8df4a6e61af" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 } } peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } } }
I20240502 08:17:21.647284 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.668371 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.668751 8295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.680263 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.681952 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.690963 8165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.713403 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.744650 11925 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.818902 11944 consensus_queue.cc:237] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } }
I20240502 08:17:21.819696 12106 raft_consensus.cc:2932] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 89ae15f3455446339957f9559f3857da (127.6.5.2) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } } }
I20240502 08:17:21.823725 8041 catalog_manager.cc:5484] T c772b9342a314c3191a5caaf4395c94f P c7e49faad12d44b79209c8df4a6e61af reported cstate change: config changed from index 4 to 6, VOTER 89ae15f3455446339957f9559f3857da (127.6.5.2) evicted. New cstate: current_term: 2 leader_uuid: "c7e49faad12d44b79209c8df4a6e61af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } } }
I20240502 08:17:21.827488 8295 tablet_service.cc:1514] Processing DeleteTablet for tablet c772b9342a314c3191a5caaf4395c94f with delete_type TABLET_DATA_TOMBSTONED (TS 89ae15f3455446339957f9559f3857da not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41504
I20240502 08:17:21.828197 12373 tablet_replica.cc:330] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da: stopping tablet replica
I20240502 08:17:21.828397 12373 raft_consensus.cc:2229] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:21.828595 12373 raft_consensus.cc:2258] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:21.830070 12373 ts_tablet_manager.cc:1875] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:21.835300 12373 ts_tablet_manager.cc:1888] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:17:21.835395 12373 log.cc:1196] T c772b9342a314c3191a5caaf4395c94f P 89ae15f3455446339957f9559f3857da: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/c772b9342a314c3191a5caaf4395c94f
I20240502 08:17:21.836083 8031 catalog_manager.cc:4830] TS 89ae15f3455446339957f9559f3857da (127.6.5.2:44539): tablet c772b9342a314c3191a5caaf4395c94f (table rebalance_test_table_1 [id=37605e3d381045239bd078121ab54c76]) successfully deleted
I20240502 08:17:21.921697 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.928134 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.930666 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.944247 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.965505 8184 raft_consensus.cc:1261] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Refusing update from remote peer c7e49faad12d44b79209c8df4a6e61af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:17:21.966236 12106 consensus_queue.cc:1035] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90db359bc744f2b82b69d97627fa7b9" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 35591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:17:21.978228 8295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.980916 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:21.999815 11925 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.036581 8165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.111284 11944 consensus_queue.cc:237] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } }
I20240502 08:17:22.112004 12106 raft_consensus.cc:2932] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } } }
I20240502 08:17:22.115674 8043 catalog_manager.cc:5484] T 9802b878c2f6463983a4029b7031004d P c7e49faad12d44b79209c8df4a6e61af reported cstate change: config changed from index 4 to 6, VOTER b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1) evicted. New cstate: current_term: 2 leader_uuid: "c7e49faad12d44b79209c8df4a6e61af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c7e49faad12d44b79209c8df4a6e61af" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 33883 } } }
I20240502 08:17:22.118865 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet 9802b878c2f6463983a4029b7031004d with delete_type TABLET_DATA_TOMBSTONED (TS b90db359bc744f2b82b69d97627fa7b9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46030
I20240502 08:17:22.119503 12427 tablet_replica.cc:330] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9: stopping tablet replica
I20240502 08:17:22.119666 12427 raft_consensus.cc:2229] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:22.119835 12427 raft_consensus.cc:2258] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:22.121126 12427 ts_tablet_manager.cc:1875] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:22.125245 12427 ts_tablet_manager.cc:1888] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:17:22.125358 12427 log.cc:1196] T 9802b878c2f6463983a4029b7031004d P b90db359bc744f2b82b69d97627fa7b9: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/9802b878c2f6463983a4029b7031004d
I20240502 08:17:22.126039 8031 catalog_manager.cc:4830] TS b90db359bc744f2b82b69d97627fa7b9 (127.6.5.1:35591): tablet 9802b878c2f6463983a4029b7031004d (table rebalance_test_table_3 [id=d999095ea3574127ad81d93a2c85019d]) successfully deleted
I20240502 08:17:22.420939 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.427170 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.434756 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.437386 8295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.447115 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.492264 8165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.528661 11925 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.543913 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.674278 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.674813 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.686350 8295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.691394 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.707012 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.719393 8165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.734493 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:22.779254 11925 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:23.038122 9373 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:23.046422 9775 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:23.047583 9905 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:23.055212 8295 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:23.067945 9505 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:23.112038 8165 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:23.115480 9645 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:23.132113 11925 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+------------------+---------
1e7a91bc1fb549619d016b96f81b6ff5 | 127.6.5.62:44177 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+------------------+-------------------------
builtin_ntp_servers | 127.6.5.20:43353 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-----------------+---------+----------+----------------+-----------------
00b1227597554ea18f7e125de6d2760e | 127.6.5.3:43567 | HEALTHY | <none> | 2 | 0
646203131e6c42a2a2e631374efbe5ee | 127.6.5.6:34035 | HEALTHY | <none> | 2 | 0
6e2d5eaefade459c8e1fce0db41f2aca | 127.6.5.7:39789 | HEALTHY | <none> | 2 | 0
89ae15f3455446339957f9559f3857da | 127.6.5.2:44539 | HEALTHY | <none> | 2 | 0
b58bd494983a4a2885cda3387b1be902 | 127.6.5.4:35529 | HEALTHY | <none> | 2 | 0
b90db359bc744f2b82b69d97627fa7b9 | 127.6.5.1:35591 | HEALTHY | <none> | 2 | 0
c4e97cd51e9949339f937b6be4cefebb | 127.6.5.5:34875 | HEALTHY | <none> | 2 | 0
c7e49faad12d44b79209c8df4a6e61af | 127.6.5.8:33883 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.6.5.1 | experimental | 127.6.5.1:35591
local_ip_for_outbound_sockets | 127.6.5.2 | experimental | 127.6.5.2:44539
local_ip_for_outbound_sockets | 127.6.5.3 | experimental | 127.6.5.3:43567
local_ip_for_outbound_sockets | 127.6.5.4 | experimental | 127.6.5.4:35529
local_ip_for_outbound_sockets | 127.6.5.5 | experimental | 127.6.5.5:34875
local_ip_for_outbound_sockets | 127.6.5.6 | experimental | 127.6.5.6:34035
local_ip_for_outbound_sockets | 127.6.5.7 | experimental | 127.6.5.7:39789
local_ip_for_outbound_sockets | 127.6.5.8 | experimental | 127.6.5.8:33883
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.6.5.1:35591
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.6.5.2:44539
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.6.5.3:43567
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.6.5.4:35529
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.6.5.5:34875
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.6.5.6:34035
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.6.5.7:39789
server_dump_info_path | /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.6.5.8:33883
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+------------------+-------------------------
builtin_ntp_servers | 127.6.5.20:43353 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 08:17:23.176613 6164 log_verifier.cc:126] Checking tablet 0b3e2b0c939e4182bb851b4cdc802301
I20240502 08:17:23.187465 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0b3e2b0c939e4182bb851b4cdc802301
I20240502 08:17:23.187579 6164 log_verifier.cc:126] Checking tablet 1fabef91cc4d4df3abfe3bdb393790e8
I20240502 08:17:23.197212 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 1fabef91cc4d4df3abfe3bdb393790e8
I20240502 08:17:23.197288 6164 log_verifier.cc:126] Checking tablet 3d96e8d4b891410980ae6a696799a60b
I20240502 08:17:23.206686 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3d96e8d4b891410980ae6a696799a60b
I20240502 08:17:23.206761 6164 log_verifier.cc:126] Checking tablet 4a905a7206774ff096194077781dcd45
I20240502 08:17:23.215668 6164 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4a905a7206774ff096194077781dcd45
I20240502 08:17:23.215739 6164 log_verifier.cc:126] Checking tablet 72b8cb6631624a839ec2b3a77abaa1df
I20240502 08:17:23.224956 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 72b8cb6631624a839ec2b3a77abaa1df
I20240502 08:17:23.225025 6164 log_verifier.cc:126] Checking tablet 74a88bd56f86453280555608afcb98f3
I20240502 08:17:23.234308 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 74a88bd56f86453280555608afcb98f3
I20240502 08:17:23.234375 6164 log_verifier.cc:126] Checking tablet 9802b878c2f6463983a4029b7031004d
I20240502 08:17:23.246302 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 9802b878c2f6463983a4029b7031004d
I20240502 08:17:23.246376 6164 log_verifier.cc:126] Checking tablet aa41e5f5e3ef4dec89a0a39198f7a53c
I20240502 08:17:23.255756 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet aa41e5f5e3ef4dec89a0a39198f7a53c
I20240502 08:17:23.255836 6164 log_verifier.cc:126] Checking tablet afc5bc8efa7d4a3fb8970bf100c754a2
I20240502 08:17:23.265583 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet afc5bc8efa7d4a3fb8970bf100c754a2
I20240502 08:17:23.265651 6164 log_verifier.cc:126] Checking tablet bdce98c845724dc3a9cedf89e3bec62c
I20240502 08:17:23.275126 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet bdce98c845724dc3a9cedf89e3bec62c
I20240502 08:17:23.275193 6164 log_verifier.cc:126] Checking tablet c772b9342a314c3191a5caaf4395c94f
I20240502 08:17:23.287057 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c772b9342a314c3191a5caaf4395c94f
I20240502 08:17:23.287133 6164 log_verifier.cc:126] Checking tablet d9a643e4d8dc4eeba0a31fc105c141a0
I20240502 08:17:23.295914 6164 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d9a643e4d8dc4eeba0a31fc105c141a0
I20240502 08:17:23.295980 6164 log_verifier.cc:126] Checking tablet da122388bec1424bb9e07919dd2d3cf0
I20240502 08:17:23.300838 6164 log_verifier.cc:177] Verified matching terms for 2 ops in tablet da122388bec1424bb9e07919dd2d3cf0
I20240502 08:17:23.300904 6164 log_verifier.cc:126] Checking tablet dba45a53e0ea4de1b03d12e36a8bb7bb
I20240502 08:17:23.309464 6164 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dba45a53e0ea4de1b03d12e36a8bb7bb
I20240502 08:17:23.309533 6164 log_verifier.cc:126] Checking tablet f766891d6cea4e1e827d3233c2e17a27
I20240502 08:17:23.322387 6164 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f766891d6cea4e1e827d3233c2e17a27
I20240502 08:17:23.322496 6164 log_verifier.cc:126] Checking tablet f87d008812d847bb89dcf4034387bb51
I20240502 08:17:23.334270 6164 log_verifier.cc:177] Verified matching terms for 6 ops in tablet f87d008812d847bb89dcf4034387bb51
I20240502 08:17:23.335136 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 8103
I20240502 08:17:23.336588 8224 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:23.529218 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 8103
I20240502 08:17:23.557659 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 8233
I20240502 08:17:23.558724 8355 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:23.750957 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 8233
I20240502 08:17:23.777535 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 9311
I20240502 08:17:23.783988 9432 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:23.964601 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 9311
I20240502 08:17:23.988446 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 9442
I20240502 08:17:23.994215 9566 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:24.164718 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 9442
I20240502 08:17:24.187719 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 9574
I20240502 08:17:24.193554 9705 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:24.366935 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 9574
I20240502 08:17:24.390388 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 9713
I20240502 08:17:24.396335 9835 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:24.563091 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 9713
W20240502 08:17:24.586721 8030 connection.cc:535] server connection from 127.6.5.6:45357 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:24.586916 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 9843
I20240502 08:17:24.596712 9965 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:24.765935 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 9843
I20240502 08:17:24.788956 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 11863
I20240502 08:17:24.794610 11985 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:24.906481 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 11863
I20240502 08:17:24.922834 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 8013
I20240502 08:17:24.923915 8074 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:25.142855 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 8013
2024-05-02T08:17:25Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (40813 ms)
[----------] 1 test from RebalanceParamTest (40813 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240502 08:17:25.209942 6164 test_util.cc:268] Using random seed: 1001300794
I20240502 08:17:25.211797 6164 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:17:25.211858 6164 ts_itest-base.cc:113] --------------
I20240502 08:17:25.211899 6164 ts_itest-base.cc:114] 8 tablet servers
I20240502 08:17:25.211954 6164 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:17:25.212015 6164 ts_itest-base.cc:116] --------------
2024-05-02T08:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:17:25Z Disabled control of system clock
I20240502 08:17:25.237006 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.5.62:46027
--webserver_interface=127.6.5.62
--webserver_port=0
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.5.62:46027
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 08:17:25.547400 12557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:25.547684 12557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:25.547780 12557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:25.554757 12557 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:25.554865 12557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:17:25.554919 12557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:25.554973 12557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:17:25.555022 12557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:17:25.563421 12557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.5.62:46027
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.5.62:46027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.5.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:25.564491 12557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:25.566056 12557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:25.576572 12563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:25.577162 12562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:25.577494 12565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:25.578086 12557 server_base.cc:1034] running on GCE node
I20240502 08:17:25.578785 12557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:25.580075 12557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:25.581290 12557 hybrid_clock.cc:648] HybridClock initialized: now 1714637845581245 us; error 29 us; skew 500 ppm
I20240502 08:17:25.581617 12557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:25.583743 12557 webserver.cc:460] Webserver started at http://127.6.5.62:37553/ using document root <none> and password file <none>
I20240502 08:17:25.584424 12557 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:25.584532 12557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:25.584831 12557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:25.587285 12557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "17be716fc99c4d588d7726089a2116d2"
format_stamp: "Formatted at 2024-05-02 08:17:25 on dist-test-slave-h7h6"
I20240502 08:17:25.588032 12557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "17be716fc99c4d588d7726089a2116d2"
format_stamp: "Formatted at 2024-05-02 08:17:25 on dist-test-slave-h7h6"
I20240502 08:17:25.592753 12557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 08:17:25.595844 12571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:25.597010 12557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:25.597162 12557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
uuid: "17be716fc99c4d588d7726089a2116d2"
format_stamp: "Formatted at 2024-05-02 08:17:25 on dist-test-slave-h7h6"
I20240502 08:17:25.597343 12557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:25.627125 12557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:25.627878 12557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:25.628137 12557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:25.643836 12557 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.62:46027
I20240502 08:17:25.643843 12623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.62:46027 every 8 connection(s)
I20240502 08:17:25.645011 12557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:17:25.648365 12624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:25.650859 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 12557
I20240502 08:17:25.651038 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:17:25.658943 12624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Bootstrap starting.
I20240502 08:17:25.662263 12624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:25.663327 12624 log.cc:824] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:25.665711 12624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: No bootstrap required, opened a new log
I20240502 08:17:25.671382 12624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17be716fc99c4d588d7726089a2116d2" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 46027 } }
I20240502 08:17:25.671757 12624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:25.671869 12624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17be716fc99c4d588d7726089a2116d2, State: Initialized, Role: FOLLOWER
I20240502 08:17:25.672402 12624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17be716fc99c4d588d7726089a2116d2" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 46027 } }
I20240502 08:17:25.672574 12624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:17:25.672667 12624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:17:25.672780 12624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:25.674620 12624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17be716fc99c4d588d7726089a2116d2" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 46027 } }
I20240502 08:17:25.675027 12624 leader_election.cc:302] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17be716fc99c4d588d7726089a2116d2; no voters:
I20240502 08:17:25.675490 12624 leader_election.cc:288] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:17:25.675650 12629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:25.676180 12629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [term 1 LEADER]: Becoming Leader. State: Replica: 17be716fc99c4d588d7726089a2116d2, State: Running, Role: LEADER
I20240502 08:17:25.676649 12629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17be716fc99c4d588d7726089a2116d2" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 46027 } }
I20240502 08:17:25.677196 12624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:17:25.679607 12630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17be716fc99c4d588d7726089a2116d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17be716fc99c4d588d7726089a2116d2" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 46027 } } }
I20240502 08:17:25.679852 12631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17be716fc99c4d588d7726089a2116d2. Latest consensus state: current_term: 1 leader_uuid: "17be716fc99c4d588d7726089a2116d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17be716fc99c4d588d7726089a2116d2" member_type: VOTER last_known_addr { host: "127.6.5.62" port: 46027 } } }
I20240502 08:17:25.680032 12630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:25.680112 12631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:25.681332 12635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:17:25.685230 12635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:17:25.693542 12635 catalog_manager.cc:1347] Generated new cluster ID: 01542c1ba231476592035b2cbae3ef43
I20240502 08:17:25.693641 12635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:17:25.713620 12635 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:17:25.715261 12635 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:17:25.731930 12635 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Generated new TSK 0
I20240502 08:17:25.732656 12635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:17:25.738766 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.1:0
--local_ip_for_outbound_sockets=127.6.5.1
--webserver_interface=127.6.5.1
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:17:26.044628 12648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:26.044872 12648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:26.044979 12648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:26.052184 12648 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:26.052296 12648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:26.052445 12648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.1
I20240502 08:17:26.061086 12648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.5.1
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.1
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:26.062225 12648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:26.063884 12648 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:26.075554 12653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:26.076133 12654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:26.076539 12656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:26.077028 12648 server_base.cc:1034] running on GCE node
I20240502 08:17:26.077589 12648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:26.078516 12648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:26.079713 12648 hybrid_clock.cc:648] HybridClock initialized: now 1714637846079664 us; error 42 us; skew 500 ppm
I20240502 08:17:26.079993 12648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:26.082093 12648 webserver.cc:460] Webserver started at http://127.6.5.1:42951/ using document root <none> and password file <none>
I20240502 08:17:26.082849 12648 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:26.082962 12648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:26.083257 12648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:26.085575 12648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "29b035434087435cbaf4ebcac98eedee"
format_stamp: "Formatted at 2024-05-02 08:17:26 on dist-test-slave-h7h6"
I20240502 08:17:26.086287 12648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "29b035434087435cbaf4ebcac98eedee"
format_stamp: "Formatted at 2024-05-02 08:17:26 on dist-test-slave-h7h6"
I20240502 08:17:26.090996 12648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 08:17:26.094058 12662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:26.095366 12648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:26.095530 12648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "29b035434087435cbaf4ebcac98eedee"
format_stamp: "Formatted at 2024-05-02 08:17:26 on dist-test-slave-h7h6"
I20240502 08:17:26.095710 12648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:26.138563 12648 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:26.139324 12648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:26.139577 12648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:26.140450 12648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:26.141839 12648 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:26.141932 12648 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:26.142036 12648 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:26.142107 12648 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:26.172024 12648 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.1:33183
I20240502 08:17:26.172098 12775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.1:33183 every 8 connection(s)
I20240502 08:17:26.173264 12648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:17:26.177866 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 12648
I20240502 08:17:26.178061 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:17:26.181576 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.2:0
--local_ip_for_outbound_sockets=127.6.5.2
--webserver_interface=127.6.5.2
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:26.185976 12776 heartbeater.cc:347] Connected to a master server at 127.6.5.62:46027
I20240502 08:17:26.186257 12776 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:26.187047 12776 heartbeater.cc:511] Master 127.6.5.62:46027 requested a full tablet report, sending...
I20240502 08:17:26.188915 12588 ts_manager.cc:194] Registered new tserver with Master: 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:26.190260 12588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.1:39033
W20240502 08:17:26.490922 12780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:26.491186 12780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:26.491333 12780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:26.498718 12780 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:26.498831 12780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:26.498975 12780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.2
I20240502 08:17:26.507479 12780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.5.2
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.2
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:26.508551 12780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:26.510103 12780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:26.521690 12785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:26.522137 12788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:26.522557 12786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:26.523128 12780 server_base.cc:1034] running on GCE node
I20240502 08:17:26.523676 12780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:26.524524 12780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:26.525727 12780 hybrid_clock.cc:648] HybridClock initialized: now 1714637846525680 us; error 35 us; skew 500 ppm
I20240502 08:17:26.526038 12780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:26.528110 12780 webserver.cc:460] Webserver started at http://127.6.5.2:43309/ using document root <none> and password file <none>
I20240502 08:17:26.528764 12780 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:26.528895 12780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:26.529193 12780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:26.531757 12780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e1dd14f7ca764a1490df419f13f2804a"
format_stamp: "Formatted at 2024-05-02 08:17:26 on dist-test-slave-h7h6"
I20240502 08:17:26.532547 12780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e1dd14f7ca764a1490df419f13f2804a"
format_stamp: "Formatted at 2024-05-02 08:17:26 on dist-test-slave-h7h6"
I20240502 08:17:26.537506 12780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 08:17:26.540741 12794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:26.541968 12780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:26.542115 12780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e1dd14f7ca764a1490df419f13f2804a"
format_stamp: "Formatted at 2024-05-02 08:17:26 on dist-test-slave-h7h6"
I20240502 08:17:26.542292 12780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:26.565804 12780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:26.566535 12780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:26.566772 12780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:26.567616 12780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:26.568924 12780 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:26.569015 12780 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:26.569118 12780 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:26.569185 12780 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:26.597368 12780 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.2:36933
I20240502 08:17:26.597424 12906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.2:36933 every 8 connection(s)
I20240502 08:17:26.598522 12780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:17:26.601305 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 12780
I20240502 08:17:26.601467 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:17:26.604950 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.3:0
--local_ip_for_outbound_sockets=127.6.5.3
--webserver_interface=127.6.5.3
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:26.612519 12907 heartbeater.cc:347] Connected to a master server at 127.6.5.62:46027
I20240502 08:17:26.612793 12907 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:26.613559 12907 heartbeater.cc:511] Master 127.6.5.62:46027 requested a full tablet report, sending...
I20240502 08:17:26.615041 12588 ts_manager.cc:194] Registered new tserver with Master: e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:26.615898 12588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.2:51077
W20240502 08:17:26.909744 12910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:26.910003 12910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:26.910126 12910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:26.917176 12910 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:26.917271 12910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:26.917414 12910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.3
I20240502 08:17:26.925820 12910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.5.3
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.3
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:26.926914 12910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:26.928520 12910 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:26.939407 12915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:26.939931 12916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:26.940330 12918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:26.940874 12910 server_base.cc:1034] running on GCE node
I20240502 08:17:26.941346 12910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:26.942106 12910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:26.943289 12910 hybrid_clock.cc:648] HybridClock initialized: now 1714637846943241 us; error 37 us; skew 500 ppm
I20240502 08:17:26.943565 12910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:26.945541 12910 webserver.cc:460] Webserver started at http://127.6.5.3:40219/ using document root <none> and password file <none>
I20240502 08:17:26.946208 12910 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:26.946310 12910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:26.946631 12910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:26.948944 12910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "60b6e265e7754b7187307d05ef405dd4"
format_stamp: "Formatted at 2024-05-02 08:17:26 on dist-test-slave-h7h6"
I20240502 08:17:26.949612 12910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "60b6e265e7754b7187307d05ef405dd4"
format_stamp: "Formatted at 2024-05-02 08:17:26 on dist-test-slave-h7h6"
I20240502 08:17:26.954048 12910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 08:17:26.956974 12924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:26.958120 12910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:26.958250 12910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "60b6e265e7754b7187307d05ef405dd4"
format_stamp: "Formatted at 2024-05-02 08:17:26 on dist-test-slave-h7h6"
I20240502 08:17:26.958415 12910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:26.972335 12910 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:26.972980 12910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:26.973201 12910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:26.974031 12910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:26.975405 12910 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:26.975486 12910 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:26.975558 12910 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:26.975610 12910 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:27.003041 12910 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.3:46757
I20240502 08:17:27.003108 13036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.3:46757 every 8 connection(s)
I20240502 08:17:27.004104 12910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:17:27.004506 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 12910
I20240502 08:17:27.004683 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:17:27.008159 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.4:0
--local_ip_for_outbound_sockets=127.6.5.4
--webserver_interface=127.6.5.4
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:27.016979 13037 heartbeater.cc:347] Connected to a master server at 127.6.5.62:46027
I20240502 08:17:27.017237 13037 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:27.017894 13037 heartbeater.cc:511] Master 127.6.5.62:46027 requested a full tablet report, sending...
I20240502 08:17:27.019135 12588 ts_manager.cc:194] Registered new tserver with Master: 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:27.019896 12588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.3:34399
I20240502 08:17:27.192694 12776 heartbeater.cc:503] Master 127.6.5.62:46027 was elected leader, sending a full tablet report...
W20240502 08:17:27.316124 13040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:27.316395 13040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:27.316524 13040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:27.323642 13040 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:27.323755 13040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:27.323909 13040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.4
I20240502 08:17:27.332209 13040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.5.4
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.4
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:27.333302 13040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:27.334890 13040 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:27.346016 13045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:27.346547 13046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:27.346941 13048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:27.347468 13040 server_base.cc:1034] running on GCE node
I20240502 08:17:27.347959 13040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:27.348742 13040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:27.349925 13040 hybrid_clock.cc:648] HybridClock initialized: now 1714637847349882 us; error 32 us; skew 500 ppm
I20240502 08:17:27.350211 13040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:27.352254 13040 webserver.cc:460] Webserver started at http://127.6.5.4:36537/ using document root <none> and password file <none>
I20240502 08:17:27.352910 13040 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:27.353017 13040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:27.353303 13040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:27.355710 13040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
format_stamp: "Formatted at 2024-05-02 08:17:27 on dist-test-slave-h7h6"
I20240502 08:17:27.356423 13040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
format_stamp: "Formatted at 2024-05-02 08:17:27 on dist-test-slave-h7h6"
I20240502 08:17:27.360961 13040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240502 08:17:27.364163 13054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:27.365356 13040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:27.365500 13040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
format_stamp: "Formatted at 2024-05-02 08:17:27 on dist-test-slave-h7h6"
I20240502 08:17:27.365670 13040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:27.383762 13040 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:27.384457 13040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:27.384693 13040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:27.385514 13040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:27.387028 13040 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:27.387138 13040 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:27.387257 13040 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:27.387341 13040 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:27.416311 13040 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.4:32979
I20240502 08:17:27.416376 13166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.4:32979 every 8 connection(s)
I20240502 08:17:27.417431 13040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:17:27.427346 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 13040
I20240502 08:17:27.427520 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:17:27.428406 13167 heartbeater.cc:347] Connected to a master server at 127.6.5.62:46027
I20240502 08:17:27.428682 13167 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:27.429497 13167 heartbeater.cc:511] Master 127.6.5.62:46027 requested a full tablet report, sending...
I20240502 08:17:27.430773 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.5:0
--local_ip_for_outbound_sockets=127.6.5.5
--webserver_interface=127.6.5.5
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:27.430835 12588 ts_manager.cc:194] Registered new tserver with Master: 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:27.431582 12588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.4:48899
I20240502 08:17:27.618381 12907 heartbeater.cc:503] Master 127.6.5.62:46027 was elected leader, sending a full tablet report...
W20240502 08:17:27.737558 13170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:27.737821 13170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:27.737948 13170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:27.744947 13170 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:27.745062 13170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:27.745208 13170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.5
I20240502 08:17:27.753623 13170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.5.5
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.5
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:27.754716 13170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:27.756258 13170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:27.767457 13175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:27.767920 13178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:27.768374 13176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:27.768939 13170 server_base.cc:1034] running on GCE node
I20240502 08:17:27.769460 13170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:27.770319 13170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:27.771515 13170 hybrid_clock.cc:648] HybridClock initialized: now 1714637847771468 us; error 42 us; skew 500 ppm
I20240502 08:17:27.771813 13170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:27.773808 13170 webserver.cc:460] Webserver started at http://127.6.5.5:37259/ using document root <none> and password file <none>
I20240502 08:17:27.774484 13170 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:27.774596 13170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:27.774888 13170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:27.777215 13170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "db7893bb29a047b3b82cc60bfd8930a3"
format_stamp: "Formatted at 2024-05-02 08:17:27 on dist-test-slave-h7h6"
I20240502 08:17:27.777943 13170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "db7893bb29a047b3b82cc60bfd8930a3"
format_stamp: "Formatted at 2024-05-02 08:17:27 on dist-test-slave-h7h6"
I20240502 08:17:27.782516 13170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20240502 08:17:27.785480 13184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:27.786708 13170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:27.786863 13170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "db7893bb29a047b3b82cc60bfd8930a3"
format_stamp: "Formatted at 2024-05-02 08:17:27 on dist-test-slave-h7h6"
I20240502 08:17:27.787029 13170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:27.799266 13170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:27.799947 13170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:27.800186 13170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:27.801098 13170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:27.802495 13170 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:27.802588 13170 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:27.802687 13170 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:27.802757 13170 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:27.830986 13170 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.5:44131
I20240502 08:17:27.831044 13296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.5:44131 every 8 connection(s)
I20240502 08:17:27.832077 13170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:17:27.838256 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 13170
I20240502 08:17:27.838424 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:17:27.841527 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.6:0
--local_ip_for_outbound_sockets=127.6.5.6
--webserver_interface=127.6.5.6
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:27.843438 13297 heartbeater.cc:347] Connected to a master server at 127.6.5.62:46027
I20240502 08:17:27.843672 13297 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:27.844286 13297 heartbeater.cc:511] Master 127.6.5.62:46027 requested a full tablet report, sending...
I20240502 08:17:27.845501 12588 ts_manager.cc:194] Registered new tserver with Master: db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:17:27.846419 12588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.5:59963
I20240502 08:17:28.021983 13037 heartbeater.cc:503] Master 127.6.5.62:46027 was elected leader, sending a full tablet report...
W20240502 08:17:28.157687 13300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:28.157953 13300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:28.158229 13300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:28.165786 13300 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:28.165910 13300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:28.166066 13300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.6
I20240502 08:17:28.176290 13300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.5.6
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.6
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:28.177515 13300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:28.179183 13300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:28.190512 13305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:28.190984 13308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:28.191468 13306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:28.192025 13300 server_base.cc:1034] running on GCE node
I20240502 08:17:28.192503 13300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:28.193286 13300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:28.194491 13300 hybrid_clock.cc:648] HybridClock initialized: now 1714637848194413 us; error 48 us; skew 500 ppm
I20240502 08:17:28.194757 13300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:28.196935 13300 webserver.cc:460] Webserver started at http://127.6.5.6:45579/ using document root <none> and password file <none>
I20240502 08:17:28.197593 13300 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:28.197691 13300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:28.197986 13300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:28.200403 13300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
format_stamp: "Formatted at 2024-05-02 08:17:28 on dist-test-slave-h7h6"
I20240502 08:17:28.201107 13300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
format_stamp: "Formatted at 2024-05-02 08:17:28 on dist-test-slave-h7h6"
I20240502 08:17:28.205718 13300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 08:17:28.209120 13314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:28.210525 13300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:28.210691 13300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
format_stamp: "Formatted at 2024-05-02 08:17:28 on dist-test-slave-h7h6"
I20240502 08:17:28.210901 13300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:28.232072 13300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:28.232797 13300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:28.233050 13300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:28.233893 13300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:28.235271 13300 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:28.235356 13300 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:28.235455 13300 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:28.235517 13300 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:28.261328 13300 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.6:44983
I20240502 08:17:28.261384 13426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.6:44983 every 8 connection(s)
I20240502 08:17:28.262570 13300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:17:28.271044 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 13300
I20240502 08:17:28.271219 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 08:17:28.275039 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.7:0
--local_ip_for_outbound_sockets=127.6.5.7
--webserver_interface=127.6.5.7
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:28.279539 13427 heartbeater.cc:347] Connected to a master server at 127.6.5.62:46027
I20240502 08:17:28.279781 13427 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:28.280418 13427 heartbeater.cc:511] Master 127.6.5.62:46027 requested a full tablet report, sending...
I20240502 08:17:28.281807 12588 ts_manager.cc:194] Registered new tserver with Master: 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:17:28.282750 12588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.6:44531
I20240502 08:17:28.434010 13167 heartbeater.cc:503] Master 127.6.5.62:46027 was elected leader, sending a full tablet report...
W20240502 08:17:28.579313 13430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:28.579545 13430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:28.579664 13430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:28.586643 13430 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:28.586741 13430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:28.586902 13430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.7
I20240502 08:17:28.595357 13430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.5.7
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.7
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:28.596423 13430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:28.598021 13430 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:28.609840 13436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:28.610296 13435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:28.610821 13438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:28.611308 13430 server_base.cc:1034] running on GCE node
I20240502 08:17:28.611897 13430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:28.612937 13430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:28.614118 13430 hybrid_clock.cc:648] HybridClock initialized: now 1714637848614079 us; error 31 us; skew 500 ppm
I20240502 08:17:28.614395 13430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:28.616498 13430 webserver.cc:460] Webserver started at http://127.6.5.7:41899/ using document root <none> and password file <none>
I20240502 08:17:28.617151 13430 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:28.617251 13430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:28.617535 13430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:28.619864 13430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "9333cb10227b4801b4fa7c24e22e131b"
format_stamp: "Formatted at 2024-05-02 08:17:28 on dist-test-slave-h7h6"
I20240502 08:17:28.620684 13430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "9333cb10227b4801b4fa7c24e22e131b"
format_stamp: "Formatted at 2024-05-02 08:17:28 on dist-test-slave-h7h6"
I20240502 08:17:28.625577 13430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 08:17:28.628710 13444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:28.629899 13430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:28.630051 13430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9333cb10227b4801b4fa7c24e22e131b"
format_stamp: "Formatted at 2024-05-02 08:17:28 on dist-test-slave-h7h6"
I20240502 08:17:28.630226 13430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:28.656044 13430 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:28.656735 13430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:28.656975 13430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:28.657795 13430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:28.659157 13430 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:28.659255 13430 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:28.659359 13430 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:28.659430 13430 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:28.688488 13430 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.7:38851
I20240502 08:17:28.688573 13556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.7:38851 every 8 connection(s)
I20240502 08:17:28.689626 13430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:17:28.694839 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 13430
I20240502 08:17:28.695016 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 08:17:28.698266 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.8:0
--local_ip_for_outbound_sockets=127.6.5.8
--webserver_interface=127.6.5.8
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:17:28.702438 13557 heartbeater.cc:347] Connected to a master server at 127.6.5.62:46027
I20240502 08:17:28.702776 13557 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:28.703462 13557 heartbeater.cc:511] Master 127.6.5.62:46027 requested a full tablet report, sending...
I20240502 08:17:28.704820 12588 ts_manager.cc:194] Registered new tserver with Master: 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:17:28.705560 12588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.7:52913
I20240502 08:17:28.849011 13297 heartbeater.cc:503] Master 127.6.5.62:46027 was elected leader, sending a full tablet report...
W20240502 08:17:29.005015 13560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:29.005277 13560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:29.005426 13560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:29.012845 13560 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:17:29.012952 13560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:29.013075 13560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.8
I20240502 08:17:29.021800 13560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.5.8
--webserver_port=0
--tserver_master_addrs=127.6.5.62:46027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.8
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:29.022991 13560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:29.024698 13560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:29.036388 13565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:29.036840 13566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:29.037281 13568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:29.042273 13560 server_base.cc:1034] running on GCE node
I20240502 08:17:29.042819 13560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:29.043597 13560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:29.044798 13560 hybrid_clock.cc:648] HybridClock initialized: now 1714637849044739 us; error 43 us; skew 500 ppm
I20240502 08:17:29.045058 13560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:29.047397 13560 webserver.cc:460] Webserver started at http://127.6.5.8:38341/ using document root <none> and password file <none>
I20240502 08:17:29.048316 13560 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:29.048439 13560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:29.048792 13560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:29.051360 13560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "10f0e36203914b85b77e78609ffced98"
format_stamp: "Formatted at 2024-05-02 08:17:29 on dist-test-slave-h7h6"
I20240502 08:17:29.052048 13560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "10f0e36203914b85b77e78609ffced98"
format_stamp: "Formatted at 2024-05-02 08:17:29 on dist-test-slave-h7h6"
I20240502 08:17:29.056594 13560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:17:29.059584 13574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:29.060715 13560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:29.060846 13560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "10f0e36203914b85b77e78609ffced98"
format_stamp: "Formatted at 2024-05-02 08:17:29 on dist-test-slave-h7h6"
I20240502 08:17:29.060997 13560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:29.079123 13560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:17:29.079766 13560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:29.079990 13560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:29.080777 13560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:29.082087 13560 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:29.082172 13560 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:29.082247 13560 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:29.082299 13560 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:17:29.111819 13560 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.8:46303
I20240502 08:17:29.111860 13686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.8:46303 every 8 connection(s)
I20240502 08:17:29.113019 13560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:17:29.116770 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 13560
I20240502 08:17:29.116948 6164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 08:17:29.125233 13687 heartbeater.cc:347] Connected to a master server at 127.6.5.62:46027
I20240502 08:17:29.125499 13687 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:29.126143 13687 heartbeater.cc:511] Master 127.6.5.62:46027 requested a full tablet report, sending...
I20240502 08:17:29.127414 12588 ts_manager.cc:194] Registered new tserver with Master: 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:17:29.128284 12588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.8:60931
I20240502 08:17:29.132571 6164 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 08:17:29.147959 6164 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 08:17:29.148092 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 13170
I20240502 08:17:29.155314 13292 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:29.233981 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 13170
I20240502 08:17:29.247151 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 13300
I20240502 08:17:29.254412 13422 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:29.330708 13427 heartbeater.cc:503] Master 127.6.5.62:46027 was elected leader, sending a full tablet report...
I20240502 08:17:29.335050 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 13300
W20240502 08:17:29.348440 12572 connection.cc:535] server connection from 127.6.5.6:44531 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:29.348590 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 13430
I20240502 08:17:29.355772 13552 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:29.433808 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 13430
I20240502 08:17:29.446866 6164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu pid 13560
I20240502 08:17:29.455011 13682 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:29.532842 6164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu with pid 13560
I20240502 08:17:33.298660 12588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:33.345963 13101 tablet_service.cc:1467] Processing CreateTablet for tablet cd8ddfa7c7a34b0c849e498ff94f8046 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.345453 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 4dffea0ca23d44c195472dbe603a4ff6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.347105 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd8ddfa7c7a34b0c849e498ff94f8046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.347644 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dffea0ca23d44c195472dbe603a4ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.349447 12842 tablet_service.cc:1467] Processing CreateTablet for tablet c0edac44fda6413d9d69d64d929532d6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.350037 12841 tablet_service.cc:1467] Processing CreateTablet for tablet cd8ddfa7c7a34b0c849e498ff94f8046 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.350834 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0edac44fda6413d9d69d64d929532d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.351331 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd8ddfa7c7a34b0c849e498ff94f8046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.351513 12709 tablet_service.cc:1467] Processing CreateTablet for tablet c0edac44fda6413d9d69d64d929532d6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.353616 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0edac44fda6413d9d69d64d929532d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.354804 12971 tablet_service.cc:1467] Processing CreateTablet for tablet c0edac44fda6413d9d69d64d929532d6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.355969 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0edac44fda6413d9d69d64d929532d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.354336 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 4dffea0ca23d44c195472dbe603a4ff6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.357358 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dffea0ca23d44c195472dbe603a4ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.352720 12708 tablet_service.cc:1467] Processing CreateTablet for tablet cd8ddfa7c7a34b0c849e498ff94f8046 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.352407 12710 tablet_service.cc:1467] Processing CreateTablet for tablet 4dffea0ca23d44c195472dbe603a4ff6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:33.362828 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd8ddfa7c7a34b0c849e498ff94f8046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.363093 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dffea0ca23d44c195472dbe603a4ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:33.378074 13718 tablet_bootstrap.cc:492] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:33.383397 13716 tablet_bootstrap.cc:492] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:33.383929 13719 tablet_bootstrap.cc:492] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:33.383989 13717 tablet_bootstrap.cc:492] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:33.386309 13719 tablet_bootstrap.cc:654] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.386612 13718 tablet_bootstrap.cc:654] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.386646 13716 tablet_bootstrap.cc:654] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.387101 13717 tablet_bootstrap.cc:654] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.387565 13719 log.cc:824] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:33.387863 13716 log.cc:824] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:33.388082 13718 log.cc:824] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:33.388233 13717 log.cc:824] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:33.395579 13716 tablet_bootstrap.cc:492] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:33.395862 13716 ts_tablet_manager.cc:1367] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240502 08:17:33.398452 13719 tablet_bootstrap.cc:492] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:33.398768 13719 ts_tablet_manager.cc:1367] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240502 08:17:33.403082 13717 tablet_bootstrap.cc:492] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:33.403340 13717 ts_tablet_manager.cc:1367] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.013s
I20240502 08:17:33.403703 13718 tablet_bootstrap.cc:492] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:33.403968 13718 ts_tablet_manager.cc:1367] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.006s
I20240502 08:17:33.405261 13716 raft_consensus.cc:348] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:33.405697 13716 raft_consensus.cc:374] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.405817 13716 raft_consensus.cc:729] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.406602 13716 consensus_queue.cc:260] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:33.409413 13719 raft_consensus.cc:348] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:33.409854 13719 raft_consensus.cc:374] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.410024 13719 raft_consensus.cc:729] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.410728 13719 consensus_queue.cc:260] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:33.411643 13717 raft_consensus.cc:348] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:33.412032 13717 raft_consensus.cc:374] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.412133 13717 raft_consensus.cc:729] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.412879 13717 consensus_queue.cc:260] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:33.413040 13718 raft_consensus.cc:348] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:33.413461 13718 raft_consensus.cc:374] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.413569 13718 raft_consensus.cc:729] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.414274 13718 consensus_queue.cc:260] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:33.414820 13716 ts_tablet_manager.cc:1398] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.019s user 0.011s sys 0.008s
I20240502 08:17:33.415304 13716 tablet_bootstrap.cc:492] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
W20240502 08:17:33.422130 13168 tablet.cc:2369] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:33.423882 13717 ts_tablet_manager.cc:1398] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.020s user 0.015s sys 0.002s
I20240502 08:17:33.425851 13717 tablet_bootstrap.cc:492] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:33.429111 13717 tablet_bootstrap.cc:654] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.430243 13716 tablet_bootstrap.cc:654] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.432399 13719 ts_tablet_manager.cc:1398] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.034s user 0.021s sys 0.008s
I20240502 08:17:33.433207 13719 tablet_bootstrap.cc:492] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:33.434557 13716 tablet_bootstrap.cc:492] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:33.434723 13716 ts_tablet_manager.cc:1367] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20240502 08:17:33.435256 13719 tablet_bootstrap.cc:654] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.435853 13716 raft_consensus.cc:348] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:33.436097 13716 raft_consensus.cc:374] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.436198 13716 raft_consensus.cc:729] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.436435 13716 consensus_queue.cc:260] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:33.437206 13716 ts_tablet_manager.cc:1398] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:33.438483 13718 ts_tablet_manager.cc:1398] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.034s user 0.021s sys 0.008s
I20240502 08:17:33.438876 13718 tablet_bootstrap.cc:492] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:33.440385 13718 tablet_bootstrap.cc:654] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.443301 13717 tablet_bootstrap.cc:492] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:33.443454 13717 ts_tablet_manager.cc:1367] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20240502 08:17:33.444324 13717 raft_consensus.cc:348] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:33.444530 13717 raft_consensus.cc:374] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.444629 13717 raft_consensus.cc:729] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.444764 13719 tablet_bootstrap.cc:492] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:33.444900 13719 ts_tablet_manager.cc:1367] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240502 08:17:33.444854 13717 consensus_queue.cc:260] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:33.445369 13717 ts_tablet_manager.cc:1398] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:33.445777 13719 raft_consensus.cc:348] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:33.445982 13719 raft_consensus.cc:374] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.446084 13719 raft_consensus.cc:729] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.446316 13718 tablet_bootstrap.cc:492] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:33.446306 13719 consensus_queue.cc:260] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:33.446449 13718 ts_tablet_manager.cc:1367] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 08:17:33.446873 13719 ts_tablet_manager.cc:1398] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:17:33.447165 13719 tablet_bootstrap.cc:492] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:33.447445 13718 raft_consensus.cc:348] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:33.447679 13718 raft_consensus.cc:374] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.447770 13718 raft_consensus.cc:729] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.447963 13718 consensus_queue.cc:260] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:33.448428 13718 ts_tablet_manager.cc:1398] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:33.448765 13719 tablet_bootstrap.cc:654] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:33.450721 13719 tablet_bootstrap.cc:492] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:33.450855 13719 ts_tablet_manager.cc:1367] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:33.451325 13719 raft_consensus.cc:348] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:33.451495 13719 raft_consensus.cc:374] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:33.451565 13719 raft_consensus.cc:729] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:33.451761 13719 consensus_queue.cc:260] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:33.452237 13719 ts_tablet_manager.cc:1398] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:33.462599 13724 raft_consensus.cc:482] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:33.462810 13724 raft_consensus.cc:504] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:33.464387 13724 leader_election.cc:288] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:33.471688 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8ddfa7c7a34b0c849e498ff94f8046" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:33.472177 12862 raft_consensus.cc:2454] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:33.472774 13056 leader_election.cc:302] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:33.473138 13724 raft_consensus.cc:2781] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:33.473258 13724 raft_consensus.cc:482] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:33.473111 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8ddfa7c7a34b0c849e498ff94f8046" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:33.473349 13724 raft_consensus.cc:3037] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.473543 12730 raft_consensus.cc:2454] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:33.475378 13724 raft_consensus.cc:504] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:33.475852 13724 leader_election.cc:288] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:33.476284 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8ddfa7c7a34b0c849e498ff94f8046" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:33.476351 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd8ddfa7c7a34b0c849e498ff94f8046" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:33.476465 12862 raft_consensus.cc:3037] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.476528 12730 raft_consensus.cc:3037] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.478585 12862 raft_consensus.cc:2454] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:33.478631 12730 raft_consensus.cc:2454] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:33.478914 13056 leader_election.cc:302] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:33.479146 13724 raft_consensus.cc:2781] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:33.479533 13724 raft_consensus.cc:686] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:33.479914 13724 consensus_queue.cc:237] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:33.483863 12587 catalog_manager.cc:5484] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:33.487053 13726 raft_consensus.cc:482] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:33.487180 13726 raft_consensus.cc:504] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:33.488445 13726 leader_election.cc:288] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:33.494333 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0edac44fda6413d9d69d64d929532d6" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:33.494560 12730 raft_consensus.cc:2454] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:33.494814 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0edac44fda6413d9d69d64d929532d6" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:33.494966 12796 leader_election.cc:302] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:33.495191 12992 raft_consensus.cc:2454] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:33.495292 13726 raft_consensus.cc:2781] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:33.495399 13726 raft_consensus.cc:482] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:33.495494 13726 raft_consensus.cc:3037] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.497231 13726 raft_consensus.cc:504] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:33.497684 13726 leader_election.cc:288] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:33.498055 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0edac44fda6413d9d69d64d929532d6" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:33.498122 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0edac44fda6413d9d69d64d929532d6" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:33.498245 12730 raft_consensus.cc:3037] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.498286 12992 raft_consensus.cc:3037] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.500775 12730 raft_consensus.cc:2454] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:33.501084 12992 raft_consensus.cc:2454] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:33.501124 12796 leader_election.cc:302] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:33.501372 13726 raft_consensus.cc:2781] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:33.501751 13726 raft_consensus.cc:686] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:17:33.502211 13726 consensus_queue.cc:237] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:33.505769 12587 catalog_manager.cc:5484] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
W20240502 08:17:33.509764 13038 tablet.cc:2369] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:33.572901 13727 raft_consensus.cc:482] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:33.573066 13727 raft_consensus.cc:504] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:33.574570 13727 leader_election.cc:288] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:33.581429 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dffea0ca23d44c195472dbe603a4ff6" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:33.581670 12730 raft_consensus.cc:2454] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:33.581749 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dffea0ca23d44c195472dbe603a4ff6" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:33.582144 13122 raft_consensus.cc:2454] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:33.582113 12926 leader_election.cc:302] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:33.582474 13727 raft_consensus.cc:2781] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:33.582599 13727 raft_consensus.cc:482] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:33.582698 13727 raft_consensus.cc:3037] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.584549 13727 raft_consensus.cc:504] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:33.585044 13727 leader_election.cc:288] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:33.585456 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dffea0ca23d44c195472dbe603a4ff6" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:33.585458 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dffea0ca23d44c195472dbe603a4ff6" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:33.585600 12730 raft_consensus.cc:3037] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.585618 13122 raft_consensus.cc:3037] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:33.587401 12730 raft_consensus.cc:2454] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:33.587577 13122 raft_consensus.cc:2454] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:33.587769 12926 leader_election.cc:302] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:33.588037 13727 raft_consensus.cc:2781] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:33.588380 13727 raft_consensus.cc:686] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:33.588829 13727 consensus_queue.cc:237] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:33.592757 12587 catalog_manager.cc:5484] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
W20240502 08:17:33.604269 12908 tablet.cc:2369] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:17:33.679003 12777 tablet.cc:2369] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:33.904987 13726 consensus_queue.cc:1035] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:33.911792 13739 consensus_queue.cc:1035] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:33.954890 13745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:33.955147 13745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:33.962167 13745 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:33.968032 13724 consensus_queue.cc:1035] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:33.974376 13735 consensus_queue.cc:1035] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 08:17:34.133323 13727 consensus_queue.cc:1035] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:34.203830 13122 raft_consensus.cc:1261] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:34.204653 13792 consensus_queue.cc:1035] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 269.444 ms
time per row: 0.134722 ms
I20240502 08:17:34.380482 12588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:34.388885 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 83e0c78cb2374ba483315ca8a827e532 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eafcd56a22d944b391206b320ed18540]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.389359 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e0c78cb2374ba483315ca8a827e532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.389401 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 83e0c78cb2374ba483315ca8a827e532 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eafcd56a22d944b391206b320ed18540]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.389757 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e0c78cb2374ba483315ca8a827e532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.390609 13102 tablet_service.cc:1467] Processing CreateTablet for tablet ea9c28b7e3b1456d96330665460a93ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=eafcd56a22d944b391206b320ed18540]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.390661 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 83e0c78cb2374ba483315ca8a827e532 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eafcd56a22d944b391206b320ed18540]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.391016 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9c28b7e3b1456d96330665460a93ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.391455 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e0c78cb2374ba483315ca8a827e532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.398818 12971 tablet_service.cc:1467] Processing CreateTablet for tablet ea9c28b7e3b1456d96330665460a93ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=eafcd56a22d944b391206b320ed18540]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.398824 12710 tablet_service.cc:1467] Processing CreateTablet for tablet 1f07b2caf1754c469a8e33382f615019 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eafcd56a22d944b391206b320ed18540]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.399181 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9c28b7e3b1456d96330665460a93ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.399150 12841 tablet_service.cc:1467] Processing CreateTablet for tablet ea9c28b7e3b1456d96330665460a93ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=eafcd56a22d944b391206b320ed18540]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.399330 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f07b2caf1754c469a8e33382f615019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.399518 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9c28b7e3b1456d96330665460a93ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.402314 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 1f07b2caf1754c469a8e33382f615019 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eafcd56a22d944b391206b320ed18540]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.402838 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f07b2caf1754c469a8e33382f615019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.403200 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 1f07b2caf1754c469a8e33382f615019 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eafcd56a22d944b391206b320ed18540]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:34.403661 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f07b2caf1754c469a8e33382f615019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:34.405560 13817 tablet_bootstrap.cc:492] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:34.407519 13817 tablet_bootstrap.cc:654] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.412340 13816 tablet_bootstrap.cc:492] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:34.413380 13818 tablet_bootstrap.cc:492] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:34.415144 13818 tablet_bootstrap.cc:654] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.415141 13816 tablet_bootstrap.cc:654] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.415704 13820 tablet_bootstrap.cc:492] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:34.417579 13820 tablet_bootstrap.cc:654] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.419261 13817 tablet_bootstrap.cc:492] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:34.419433 13817 ts_tablet_manager.cc:1367] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240502 08:17:34.420312 13817 raft_consensus.cc:348] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.420562 13817 raft_consensus.cc:374] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.420660 13817 raft_consensus.cc:729] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.420934 13817 consensus_queue.cc:260] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.421568 13817 ts_tablet_manager.cc:1398] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:34.421870 13817 tablet_bootstrap.cc:492] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:34.424912 13817 tablet_bootstrap.cc:654] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.431380 13820 tablet_bootstrap.cc:492] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:34.431380 13818 tablet_bootstrap.cc:492] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:34.431576 13820 ts_tablet_manager.cc:1367] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20240502 08:17:34.431685 13818 ts_tablet_manager.cc:1367] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20240502 08:17:34.432431 13818 raft_consensus.cc:348] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.432418 13820 raft_consensus.cc:348] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:34.432626 13818 raft_consensus.cc:374] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.432713 13818 raft_consensus.cc:729] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.432736 13820 raft_consensus.cc:374] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.432920 13820 raft_consensus.cc:729] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.432963 13818 consensus_queue.cc:260] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.433275 13820 consensus_queue.cc:260] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:34.433954 13820 ts_tablet_manager.cc:1398] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:34.435286 13818 ts_tablet_manager.cc:1398] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:17:34.435652 13817 tablet_bootstrap.cc:492] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:34.435827 13817 ts_tablet_manager.cc:1367] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20240502 08:17:34.436045 13818 tablet_bootstrap.cc:492] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:34.436584 13817 raft_consensus.cc:348] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.436798 13817 raft_consensus.cc:374] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.436878 13817 raft_consensus.cc:729] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.437114 13817 consensus_queue.cc:260] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.437907 13818 tablet_bootstrap.cc:654] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.438526 13817 ts_tablet_manager.cc:1398] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:34.438820 13817 tablet_bootstrap.cc:492] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:34.441479 13817 tablet_bootstrap.cc:654] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.446754 13818 tablet_bootstrap.cc:492] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:34.446909 13818 ts_tablet_manager.cc:1367] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240502 08:17:34.447676 13818 raft_consensus.cc:348] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.447877 13818 raft_consensus.cc:374] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.447959 13818 raft_consensus.cc:729] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.448210 13818 consensus_queue.cc:260] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.449470 13725 raft_consensus.cc:482] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:34.449765 13817 tablet_bootstrap.cc:492] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:34.449630 13725 raft_consensus.cc:504] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:34.449895 13817 ts_tablet_manager.cc:1367] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 08:17:34.450598 13817 raft_consensus.cc:348] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:34.450781 13817 raft_consensus.cc:374] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.450883 13817 raft_consensus.cc:729] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.451113 13817 consensus_queue.cc:260] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:34.451493 13818 ts_tablet_manager.cc:1398] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:34.451610 13725 leader_election.cc:288] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:34.452493 13817 ts_tablet_manager.cc:1398] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:34.454787 13816 tablet_bootstrap.cc:492] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:34.455001 13816 ts_tablet_manager.cc:1367] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.023s
I20240502 08:17:34.455816 13816 raft_consensus.cc:348] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.456077 13816 raft_consensus.cc:374] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.456188 13816 raft_consensus.cc:729] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.456432 13816 consensus_queue.cc:260] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.457516 13816 ts_tablet_manager.cc:1398] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:34.457792 13816 tablet_bootstrap.cc:492] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:34.459352 13816 tablet_bootstrap.cc:654] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.460780 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f07b2caf1754c469a8e33382f615019" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:34.461012 12862 raft_consensus.cc:2454] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:34.461537 12664 leader_election.cc:302] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:34.461915 13816 tablet_bootstrap.cc:492] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:34.461922 13725 raft_consensus.cc:2781] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:34.462025 13725 raft_consensus.cc:482] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:34.462062 13816 ts_tablet_manager.cc:1367] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:34.462105 13725 raft_consensus.cc:3037] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.462808 13816 raft_consensus.cc:348] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:34.463016 13816 raft_consensus.cc:374] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.463101 13816 raft_consensus.cc:729] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.463136 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f07b2caf1754c469a8e33382f615019" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:34.463330 13816 consensus_queue.cc:260] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:34.463572 12992 raft_consensus.cc:2454] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:34.463747 13816 ts_tablet_manager.cc:1398] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:34.464001 13816 tablet_bootstrap.cc:492] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:34.464797 13725 raft_consensus.cc:504] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:34.465319 13725 leader_election.cc:288] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:34.465626 13816 tablet_bootstrap.cc:654] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:34.466058 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f07b2caf1754c469a8e33382f615019" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:34.466065 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f07b2caf1754c469a8e33382f615019" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:34.466233 12862 raft_consensus.cc:3037] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.466285 12992 raft_consensus.cc:3037] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.467847 13816 tablet_bootstrap.cc:492] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:34.467998 13816 ts_tablet_manager.cc:1367] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:34.468304 12862 raft_consensus.cc:2454] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:34.468612 12664 leader_election.cc:302] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:34.468845 13725 raft_consensus.cc:2781] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:34.468807 13816 raft_consensus.cc:348] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.469034 13816 raft_consensus.cc:374] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:34.469089 12992 raft_consensus.cc:2454] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:34.469151 13816 raft_consensus.cc:729] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:34.469401 13725 raft_consensus.cc:686] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:34.469398 13816 consensus_queue.cc:260] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.469957 13816 ts_tablet_manager.cc:1398] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:34.470074 13725 consensus_queue.cc:237] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:34.472802 12581 catalog_manager.cc:5484] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:17:34.535863 13798 raft_consensus.cc:482] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:34.536024 13798 raft_consensus.cc:504] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.536825 13798 leader_election.cc:288] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:34.537251 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e0c78cb2374ba483315ca8a827e532" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:34.537546 13121 raft_consensus.cc:2454] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:34.537971 12925 leader_election.cc:302] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:34.538223 13798 raft_consensus.cc:2781] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:34.538350 13798 raft_consensus.cc:482] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:34.538475 13798 raft_consensus.cc:3037] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.540692 13798 raft_consensus.cc:504] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.541337 13798 leader_election.cc:288] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:34.541659 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e0c78cb2374ba483315ca8a827e532" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:34.541848 13121 raft_consensus.cc:3037] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.543423 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e0c78cb2374ba483315ca8a827e532" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:34.543628 12862 raft_consensus.cc:2454] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:34.543560 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83e0c78cb2374ba483315ca8a827e532" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:34.543853 12861 raft_consensus.cc:3037] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.544014 13121 raft_consensus.cc:2454] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:34.544342 12925 leader_election.cc:302] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:34.544579 13798 raft_consensus.cc:2781] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:34.544693 13798 raft_consensus.cc:686] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:34.544936 13798 consensus_queue.cc:237] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.546114 12861 raft_consensus.cc:2454] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:34.547592 12581 catalog_manager.cc:5484] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:34.731207 13798 raft_consensus.cc:482] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:34.731406 13798 raft_consensus.cc:504] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.732048 13798 leader_election.cc:288] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:34.732667 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9c28b7e3b1456d96330665460a93ac" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:34.732667 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9c28b7e3b1456d96330665460a93ac" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:34.732908 12861 raft_consensus.cc:2454] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:34.732928 13121 raft_consensus.cc:2454] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:34.733367 12925 leader_election.cc:302] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:34.733675 13798 raft_consensus.cc:2781] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:34.733776 13798 raft_consensus.cc:482] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:34.733880 13798 raft_consensus.cc:3037] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.735649 13798 raft_consensus.cc:504] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.736193 13798 leader_election.cc:288] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:34.736557 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9c28b7e3b1456d96330665460a93ac" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:34.736647 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9c28b7e3b1456d96330665460a93ac" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:34.736748 13121 raft_consensus.cc:3037] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.736881 12861 raft_consensus.cc:3037] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:34.738533 13121 raft_consensus.cc:2454] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:34.738850 12861 raft_consensus.cc:2454] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:34.738881 12925 leader_election.cc:302] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:34.739096 13798 raft_consensus.cc:2781] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:34.739248 13798 raft_consensus.cc:686] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:34.739503 13798 consensus_queue.cc:237] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:34.741839 12581 catalog_manager.cc:5484] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:34.920876 13725 consensus_queue.cc:1035] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:34.927129 13725 consensus_queue.cc:1035] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:35.049774 13798 consensus_queue.cc:1035] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:35.054117 13793 consensus_queue.cc:1035] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:35.085582 13836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:35.085850 13836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:35.092981 13836 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 08:17:35.210992 13121 raft_consensus.cc:1261] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:35.211776 13793 consensus_queue.cc:1035] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:35.213687 12862 raft_consensus.cc:1261] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:35.222043 13793 consensus_queue.cc:1035] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240502 08:17:35.241236 13873 mvcc.cc:205] Tried to move back new op lower bound from 7023156654934360064 to 7023156653014769664. Current Snapshot: MvccSnapshot[applied={T|T < 7023156654934360064}]
I20240502 08:17:35.249523 13844 mvcc.cc:205] Tried to move back new op lower bound from 7023156654934360064 to 7023156653014769664. Current Snapshot: MvccSnapshot[applied={T|T < 7023156654934360064}]
I20240502 08:17:35.286810 13878 mvcc.cc:205] Tried to move back new op lower bound from 7023156654934360064 to 7023156653014769664. Current Snapshot: MvccSnapshot[applied={T|T < 7023156654934360064 or (T in {7023156654934360064})}]
INSERT report
rows total: 2000
time total: 202.998 ms
time per row: 0.101499 ms
I20240502 08:17:35.429111 12585 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:35.437824 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 17bd47b83e7f42698a2071cf82c6aa94 (DEFAULT_TABLE table=rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.438114 12710 tablet_service.cc:1467] Processing CreateTablet for tablet 17bd47b83e7f42698a2071cf82c6aa94 (DEFAULT_TABLE table=rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.438297 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17bd47b83e7f42698a2071cf82c6aa94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.438409 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 17bd47b83e7f42698a2071cf82c6aa94 (DEFAULT_TABLE table=rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.438588 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17bd47b83e7f42698a2071cf82c6aa94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.438797 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17bd47b83e7f42698a2071cf82c6aa94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.439810 13100 tablet_service.cc:1467] Processing CreateTablet for tablet acf6ca36b79e4c69a61892d51978baa5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.440209 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf6ca36b79e4c69a61892d51978baa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.444660 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 97d0ce51e8ad46dda59ae9d14b11ac1a (DEFAULT_TABLE table=rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.445021 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d0ce51e8ad46dda59ae9d14b11ac1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.446887 13897 tablet_bootstrap.cc:492] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:35.447558 13895 tablet_bootstrap.cc:492] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:35.448477 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 97d0ce51e8ad46dda59ae9d14b11ac1a (DEFAULT_TABLE table=rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.448490 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 97d0ce51e8ad46dda59ae9d14b11ac1a (DEFAULT_TABLE table=rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.448845 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d0ce51e8ad46dda59ae9d14b11ac1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.448840 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d0ce51e8ad46dda59ae9d14b11ac1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.448987 12840 tablet_service.cc:1467] Processing CreateTablet for tablet acf6ca36b79e4c69a61892d51978baa5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.449226 13895 tablet_bootstrap.cc:654] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.449347 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf6ca36b79e4c69a61892d51978baa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.454828 12708 tablet_service.cc:1467] Processing CreateTablet for tablet acf6ca36b79e4c69a61892d51978baa5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:35.455192 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf6ca36b79e4c69a61892d51978baa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.455746 13897 tablet_bootstrap.cc:654] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.462742 13896 tablet_bootstrap.cc:492] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:35.464048 13897 tablet_bootstrap.cc:492] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:35.464195 13897 ts_tablet_manager.cc:1367] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.005s
I20240502 08:17:35.464980 13897 raft_consensus.cc:348] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.465195 13897 raft_consensus.cc:374] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.465291 13897 raft_consensus.cc:729] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.465842 13899 tablet_bootstrap.cc:492] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:35.467124 13896 tablet_bootstrap.cc:654] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.467268 13897 consensus_queue.cc:260] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.467855 13897 ts_tablet_manager.cc:1398] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:17:35.468209 13897 tablet_bootstrap.cc:492] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:35.468426 13899 tablet_bootstrap.cc:654] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.469816 13897 tablet_bootstrap.cc:654] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.470360 13895 tablet_bootstrap.cc:492] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:35.471791 13895 ts_tablet_manager.cc:1367] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.012s
I20240502 08:17:35.472568 13895 raft_consensus.cc:348] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.472779 13895 raft_consensus.cc:374] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.472862 13895 raft_consensus.cc:729] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.473078 13895 consensus_queue.cc:260] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.474694 13895 ts_tablet_manager.cc:1398] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:35.474999 13895 tablet_bootstrap.cc:492] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:35.478749 13899 tablet_bootstrap.cc:492] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:35.478741 13895 tablet_bootstrap.cc:654] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.478785 13896 tablet_bootstrap.cc:492] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:35.478920 13896 ts_tablet_manager.cc:1367] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.010s
I20240502 08:17:35.478957 13899 ts_tablet_manager.cc:1367] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20240502 08:17:35.479368 13897 tablet_bootstrap.cc:492] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:35.479519 13897 ts_tablet_manager.cc:1367] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240502 08:17:35.479653 13896 raft_consensus.cc:348] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.479871 13896 raft_consensus.cc:374] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.480031 13899 raft_consensus.cc:348] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.480284 13899 raft_consensus.cc:374] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.480427 13899 raft_consensus.cc:729] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.480736 13899 consensus_queue.cc:260] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.479959 13896 raft_consensus.cc:729] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.481170 13896 consensus_queue.cc:260] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.481776 13896 ts_tablet_manager.cc:1398] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:35.482103 13896 tablet_bootstrap.cc:492] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:35.482545 13899 ts_tablet_manager.cc:1398] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:35.483702 13896 tablet_bootstrap.cc:654] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.481698 13897 raft_consensus.cc:348] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:35.485039 13897 raft_consensus.cc:374] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.485165 13897 raft_consensus.cc:729] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.485495 13897 consensus_queue.cc:260] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:35.486593 13897 ts_tablet_manager.cc:1398] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:17:35.486925 13897 tablet_bootstrap.cc:492] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:35.488966 13895 tablet_bootstrap.cc:492] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:35.489120 13895 ts_tablet_manager.cc:1367] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.002s
I20240502 08:17:35.489212 13897 tablet_bootstrap.cc:654] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.489709 13896 tablet_bootstrap.cc:492] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:35.489866 13896 ts_tablet_manager.cc:1367] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 08:17:35.489795 13895 raft_consensus.cc:348] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:35.489995 13895 raft_consensus.cc:374] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.490078 13895 raft_consensus.cc:729] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.490324 13895 consensus_queue.cc:260] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:35.490864 13895 ts_tablet_manager.cc:1398] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:35.490561 13896 raft_consensus.cc:348] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.491120 13896 raft_consensus.cc:374] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.491150 13897 tablet_bootstrap.cc:492] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:35.491205 13896 raft_consensus.cc:729] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.491266 13897 ts_tablet_manager.cc:1367] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:35.491436 13896 consensus_queue.cc:260] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.491838 13897 raft_consensus.cc:348] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.491956 13897 raft_consensus.cc:374] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.492010 13897 raft_consensus.cc:729] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.492213 13897 consensus_queue.cc:260] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.492657 13897 ts_tablet_manager.cc:1398] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:35.492987 13896 ts_tablet_manager.cc:1398] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:35.493256 13896 tablet_bootstrap.cc:492] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:35.494897 13896 tablet_bootstrap.cc:654] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:35.496377 13896 tablet_bootstrap.cc:492] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:35.496479 13896 ts_tablet_manager.cc:1367] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:35.497001 13896 raft_consensus.cc:348] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:35.497133 13896 raft_consensus.cc:374] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:35.497217 13896 raft_consensus.cc:729] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:35.497514 13896 consensus_queue.cc:260] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:35.498030 13896 ts_tablet_manager.cc:1398] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:35.539250 13831 raft_consensus.cc:482] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:35.539368 13831 raft_consensus.cc:504] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.540000 13831 leader_election.cc:288] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:35.540428 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bd47b83e7f42698a2071cf82c6aa94" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:35.540659 12862 raft_consensus.cc:2454] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:35.541105 12664 leader_election.cc:302] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:35.541337 13831 raft_consensus.cc:2781] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:35.541466 13831 raft_consensus.cc:482] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:35.541571 13831 raft_consensus.cc:3037] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.544123 13831 raft_consensus.cc:504] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.544643 13831 leader_election.cc:288] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:35.545040 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bd47b83e7f42698a2071cf82c6aa94" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:35.545197 12862 raft_consensus.cc:3037] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.547333 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bd47b83e7f42698a2071cf82c6aa94" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:35.547358 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17bd47b83e7f42698a2071cf82c6aa94" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:35.547610 13121 raft_consensus.cc:2454] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:35.547736 12862 raft_consensus.cc:2454] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:35.548107 12664 leader_election.cc:302] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters: 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:17:35.548393 13831 raft_consensus.cc:2781] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:35.548516 13831 raft_consensus.cc:686] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:35.548784 13831 consensus_queue.cc:237] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.551038 12585 catalog_manager.cc:5484] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:35.663046 13800 raft_consensus.cc:482] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:35.663189 13800 raft_consensus.cc:504] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:35.663882 13800 leader_election.cc:288] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:35.664364 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf6ca36b79e4c69a61892d51978baa5" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:35.664618 12729 raft_consensus.cc:2454] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:35.665081 12796 leader_election.cc:302] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:35.665349 13800 raft_consensus.cc:2781] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:35.665482 13800 raft_consensus.cc:482] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:35.665593 13800 raft_consensus.cc:3037] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.668035 13800 raft_consensus.cc:504] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:35.668789 13800 leader_election.cc:288] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:35.669011 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf6ca36b79e4c69a61892d51978baa5" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:35.669193 12729 raft_consensus.cc:3037] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.670799 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf6ca36b79e4c69a61892d51978baa5" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:35.670908 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf6ca36b79e4c69a61892d51978baa5" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:35.671044 13121 raft_consensus.cc:2454] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:35.671514 12729 raft_consensus.cc:2454] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:35.671866 12796 leader_election.cc:302] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters: 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:17:35.672133 13800 raft_consensus.cc:2781] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:35.672494 13800 raft_consensus.cc:686] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:17:35.672791 13800 consensus_queue.cc:237] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:35.675181 12585 catalog_manager.cc:5484] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:35.743549 13831 raft_consensus.cc:482] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:35.743721 13831 raft_consensus.cc:504] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.744338 13831 leader_election.cc:288] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:35.744700 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d0ce51e8ad46dda59ae9d14b11ac1a" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:35.744788 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d0ce51e8ad46dda59ae9d14b11ac1a" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:35.744884 12992 raft_consensus.cc:2454] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:35.745033 13121 raft_consensus.cc:2454] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:35.745285 12664 leader_election.cc:302] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:35.745581 13831 raft_consensus.cc:2781] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:35.745677 13831 raft_consensus.cc:482] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:35.745755 13831 raft_consensus.cc:3037] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.747620 13831 raft_consensus.cc:504] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.748203 13831 leader_election.cc:288] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:35.748636 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d0ce51e8ad46dda59ae9d14b11ac1a" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:35.748634 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d0ce51e8ad46dda59ae9d14b11ac1a" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:35.748795 13121 raft_consensus.cc:3037] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.748792 12992 raft_consensus.cc:3037] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:35.750613 13121 raft_consensus.cc:2454] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:35.750609 12992 raft_consensus.cc:2454] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:35.750967 12663 leader_election.cc:302] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:35.751260 13831 raft_consensus.cc:2781] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:35.751402 13831 raft_consensus.cc:686] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:35.751631 13831 consensus_queue.cc:237] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:35.753973 12585 catalog_manager.cc:5484] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:36.001792 13121 raft_consensus.cc:3037] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:36.004756 13831 consensus_queue.cc:1035] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:36.011927 13831 consensus_queue.cc:1035] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:36.061056 13800 consensus_queue.cc:1035] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:36.065198 13120 raft_consensus.cc:3037] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:36.068775 13912 consensus_queue.cc:1035] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:36.124050 13915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:36.124333 13915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:36.131981 13915 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 08:17:36.175393 13831 consensus_queue.cc:1035] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:36.181795 13725 consensus_queue.cc:1035] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 181.293 ms
time per row: 0.0906463 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:37.301518 12587 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:37.310554 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 21f25468f1b541dc9ede27359acc2b9c (DEFAULT_TABLE table=rebalance_test_table_3 [id=f32118c718584379bfc02c6d1b29d73e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.310776 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 21f25468f1b541dc9ede27359acc2b9c (DEFAULT_TABLE table=rebalance_test_table_3 [id=f32118c718584379bfc02c6d1b29d73e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.311015 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21f25468f1b541dc9ede27359acc2b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.311211 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21f25468f1b541dc9ede27359acc2b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.311568 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 21f25468f1b541dc9ede27359acc2b9c (DEFAULT_TABLE table=rebalance_test_table_3 [id=f32118c718584379bfc02c6d1b29d73e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.311686 13102 tablet_service.cc:1467] Processing CreateTablet for tablet bf504f2522a147c198f99debdfe03d1e (DEFAULT_TABLE table=rebalance_test_table_3 [id=f32118c718584379bfc02c6d1b29d73e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.311950 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21f25468f1b541dc9ede27359acc2b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.312070 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf504f2522a147c198f99debdfe03d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.317874 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 809af1a2878d4cba9ab2f18c11a621a8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f32118c718584379bfc02c6d1b29d73e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.318251 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809af1a2878d4cba9ab2f18c11a621a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.319499 13980 tablet_bootstrap.cc:492] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:37.321099 13979 tablet_bootstrap.cc:492] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:37.322773 13980 tablet_bootstrap.cc:654] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.323194 13979 tablet_bootstrap.cc:654] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.324118 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 809af1a2878d4cba9ab2f18c11a621a8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f32118c718584379bfc02c6d1b29d73e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.324431 13981 tablet_bootstrap.cc:492] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:37.324476 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809af1a2878d4cba9ab2f18c11a621a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.325628 13981 tablet_bootstrap.cc:654] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.326982 13979 tablet_bootstrap.cc:492] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:37.327117 13979 ts_tablet_manager.cc:1367] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:17:37.327287 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 809af1a2878d4cba9ab2f18c11a621a8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f32118c718584379bfc02c6d1b29d73e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.327669 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809af1a2878d4cba9ab2f18c11a621a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.327857 13979 raft_consensus.cc:348] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.328065 13979 raft_consensus.cc:374] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.328156 13979 raft_consensus.cc:729] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.328406 13979 consensus_queue.cc:260] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.329895 13979 ts_tablet_manager.cc:1398] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:37.330847 13981 tablet_bootstrap.cc:492] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:37.331445 12840 tablet_service.cc:1467] Processing CreateTablet for tablet bf504f2522a147c198f99debdfe03d1e (DEFAULT_TABLE table=rebalance_test_table_3 [id=f32118c718584379bfc02c6d1b29d73e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.331596 13981 ts_tablet_manager.cc:1367] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 08:17:37.331794 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf504f2522a147c198f99debdfe03d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.332301 13981 raft_consensus.cc:348] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:37.332485 13981 raft_consensus.cc:374] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.332559 13981 raft_consensus.cc:729] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.332787 13981 consensus_queue.cc:260] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:37.333396 13981 ts_tablet_manager.cc:1398] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:37.337435 12970 tablet_service.cc:1467] Processing CreateTablet for tablet bf504f2522a147c198f99debdfe03d1e (DEFAULT_TABLE table=rebalance_test_table_3 [id=f32118c718584379bfc02c6d1b29d73e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:37.337769 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf504f2522a147c198f99debdfe03d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:37.343815 13979 tablet_bootstrap.cc:492] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:37.344991 13979 tablet_bootstrap.cc:654] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.345395 13981 tablet_bootstrap.cc:492] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:37.346247 13978 tablet_bootstrap.cc:492] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:37.347841 13978 tablet_bootstrap.cc:654] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.348734 13981 tablet_bootstrap.cc:654] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.349843 13980 tablet_bootstrap.cc:492] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:37.350001 13980 ts_tablet_manager.cc:1367] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.011s
I20240502 08:17:37.350801 13980 raft_consensus.cc:348] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.351009 13980 raft_consensus.cc:374] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.351105 13980 raft_consensus.cc:729] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.351356 13980 consensus_queue.cc:260] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.351682 13978 tablet_bootstrap.cc:492] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:37.351819 13978 ts_tablet_manager.cc:1367] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 08:17:37.352501 13978 raft_consensus.cc:348] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.352694 13978 raft_consensus.cc:374] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.352797 13978 raft_consensus.cc:729] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.353039 13978 consensus_queue.cc:260] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.354543 13978 ts_tablet_manager.cc:1398] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:37.354796 13978 tablet_bootstrap.cc:492] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:37.355234 13980 ts_tablet_manager.cc:1398] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:17:37.355309 13979 tablet_bootstrap.cc:492] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:37.355476 13979 ts_tablet_manager.cc:1367] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240502 08:17:37.356175 13979 raft_consensus.cc:348] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:37.356405 13979 raft_consensus.cc:374] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.356478 13979 raft_consensus.cc:729] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.356717 13979 consensus_queue.cc:260] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:37.357386 13980 tablet_bootstrap.cc:492] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:37.358558 13978 tablet_bootstrap.cc:654] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.358780 13980 tablet_bootstrap.cc:654] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.359506 13979 ts_tablet_manager.cc:1398] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:17:37.362736 13981 tablet_bootstrap.cc:492] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:37.362912 13981 ts_tablet_manager.cc:1367] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.000s
I20240502 08:17:37.363487 13981 raft_consensus.cc:348] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.363662 13981 raft_consensus.cc:374] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.363734 13981 raft_consensus.cc:729] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.363945 13981 consensus_queue.cc:260] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.364507 13981 ts_tablet_manager.cc:1398] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:37.366799 13978 tablet_bootstrap.cc:492] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:37.367005 13978 ts_tablet_manager.cc:1367] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240502 08:17:37.367063 13980 tablet_bootstrap.cc:492] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:37.367188 13980 ts_tablet_manager.cc:1367] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 08:17:37.367714 13980 raft_consensus.cc:348] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.367681 13978 raft_consensus.cc:348] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.367868 13980 raft_consensus.cc:374] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.367892 13978 raft_consensus.cc:374] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.367944 13980 raft_consensus.cc:729] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.368021 13978 raft_consensus.cc:729] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.368191 13980 consensus_queue.cc:260] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.368314 13978 consensus_queue.cc:260] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.368688 13980 ts_tablet_manager.cc:1398] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:17:37.368912 13978 ts_tablet_manager.cc:1398] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:37.369167 13978 tablet_bootstrap.cc:492] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:37.370587 13978 tablet_bootstrap.cc:654] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:37.372125 13978 tablet_bootstrap.cc:492] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:37.372246 13978 ts_tablet_manager.cc:1367] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:37.372799 13978 raft_consensus.cc:348] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:37.372941 13978 raft_consensus.cc:374] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:37.373008 13978 raft_consensus.cc:729] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:37.373204 13978 consensus_queue.cc:260] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:37.374050 13978 ts_tablet_manager.cc:1398] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:37.594444 13912 raft_consensus.cc:482] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:37.594645 13912 raft_consensus.cc:504] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.595223 13912 leader_election.cc:288] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:37.595645 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f25468f1b541dc9ede27359acc2b9c" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:37.595755 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f25468f1b541dc9ede27359acc2b9c" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:37.595921 12992 raft_consensus.cc:2454] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:37.595988 12729 raft_consensus.cc:2454] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:37.596349 12796 leader_election.cc:302] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:37.596598 13912 raft_consensus.cc:2781] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:37.596683 13912 raft_consensus.cc:482] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:37.596771 13912 raft_consensus.cc:3037] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.598743 13912 raft_consensus.cc:504] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.599328 13912 leader_election.cc:288] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:37.599647 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f25468f1b541dc9ede27359acc2b9c" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:37.599773 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f25468f1b541dc9ede27359acc2b9c" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:37.599853 12992 raft_consensus.cc:3037] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.599987 12729 raft_consensus.cc:3037] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.602541 12992 raft_consensus.cc:2454] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:37.602632 12729 raft_consensus.cc:2454] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:37.602989 12796 leader_election.cc:302] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:37.603225 13912 raft_consensus.cc:2781] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:37.603605 13912 raft_consensus.cc:686] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:17:37.603847 13912 consensus_queue.cc:237] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.606055 12582 catalog_manager.cc:5484] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:17:37.606881 13725 raft_consensus.cc:482] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:37.607013 13725 raft_consensus.cc:504] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.607558 13725 leader_election.cc:288] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:37.607986 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809af1a2878d4cba9ab2f18c11a621a8" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:37.608156 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809af1a2878d4cba9ab2f18c11a621a8" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:37.608397 13121 raft_consensus.cc:2454] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:37.608816 12663 leader_election.cc:302] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:37.609054 13725 raft_consensus.cc:2781] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:37.609164 13725 raft_consensus.cc:482] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:37.609184 12992 raft_consensus.cc:2454] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:37.609267 13725 raft_consensus.cc:3037] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.611168 13725 raft_consensus.cc:504] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.611680 13725 leader_election.cc:288] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:37.612087 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809af1a2878d4cba9ab2f18c11a621a8" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:37.612120 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809af1a2878d4cba9ab2f18c11a621a8" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:37.612272 12992 raft_consensus.cc:3037] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.612279 13121 raft_consensus.cc:3037] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.614812 13121 raft_consensus.cc:2454] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:37.614823 12992 raft_consensus.cc:2454] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:37.615180 12663 leader_election.cc:302] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:37.615478 13725 raft_consensus.cc:2781] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:37.615837 13725 raft_consensus.cc:686] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:37.616169 13725 consensus_queue.cc:237] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:37.618619 12582 catalog_manager.cc:5484] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:17:37.737804 13735 raft_consensus.cc:482] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:37.738023 13735 raft_consensus.cc:504] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:37.738809 13735 leader_election.cc:288] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:37.739962 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf504f2522a147c198f99debdfe03d1e" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:37.740190 12862 raft_consensus.cc:2454] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:37.740669 13056 leader_election.cc:302] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:37.740969 13735 raft_consensus.cc:2781] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:37.741088 13735 raft_consensus.cc:482] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:37.741201 13735 raft_consensus.cc:3037] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.743674 13735 raft_consensus.cc:504] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:37.744290 13735 leader_election.cc:288] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:37.744786 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf504f2522a147c198f99debdfe03d1e" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:37.744966 12862 raft_consensus.cc:3037] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.745282 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf504f2522a147c198f99debdfe03d1e" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:37.745548 12992 raft_consensus.cc:2454] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:37.745481 12991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf504f2522a147c198f99debdfe03d1e" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:37.745766 12991 raft_consensus.cc:3037] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:37.747745 12862 raft_consensus.cc:2454] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:37.748122 13056 leader_election.cc:302] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:37.748342 12991 raft_consensus.cc:2454] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:37.748368 13735 raft_consensus.cc:2781] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:37.748862 13735 raft_consensus.cc:686] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:37.749125 13735 consensus_queue.cc:237] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:37.752302 12582 catalog_manager.cc:5484] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:38.045465 13912 consensus_queue.cc:1035] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:38.052125 14004 consensus_queue.cc:1035] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:38.100220 14000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:38.100540 14000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:38.107715 14000 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:38.119411 13725 consensus_queue.cc:1035] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:38.124256 14016 consensus_queue.cc:1035] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240502 08:17:38.159518 13735 consensus_queue.cc:1035] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:38.165066 13998 consensus_queue.cc:1035] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 218.48 ms
time per row: 0.10924 ms
I20240502 08:17:38.465662 12586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:38.474363 12708 tablet_service.cc:1467] Processing CreateTablet for tablet afea0bbd4c0d41d4a63d33b21f542038 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.474519 12840 tablet_service.cc:1467] Processing CreateTablet for tablet afea0bbd4c0d41d4a63d33b21f542038 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.474947 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afea0bbd4c0d41d4a63d33b21f542038. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.474980 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afea0bbd4c0d41d4a63d33b21f542038. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.475622 12972 tablet_service.cc:1467] Processing CreateTablet for tablet c3d9fdffe8fb40009ca236f998ee9dba (DEFAULT_TABLE table=rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.476035 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d9fdffe8fb40009ca236f998ee9dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.476083 12841 tablet_service.cc:1467] Processing CreateTablet for tablet c3d9fdffe8fb40009ca236f998ee9dba (DEFAULT_TABLE table=rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.476433 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d9fdffe8fb40009ca236f998ee9dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.478842 13101 tablet_service.cc:1467] Processing CreateTablet for tablet afea0bbd4c0d41d4a63d33b21f542038 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.479219 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afea0bbd4c0d41d4a63d33b21f542038. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.480345 13102 tablet_service.cc:1467] Processing CreateTablet for tablet c52f670b014a4c608807fa9b8877d481 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.480675 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52f670b014a4c608807fa9b8877d481. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.482767 12709 tablet_service.cc:1467] Processing CreateTablet for tablet c3d9fdffe8fb40009ca236f998ee9dba (DEFAULT_TABLE table=rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.483158 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d9fdffe8fb40009ca236f998ee9dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.483536 12970 tablet_service.cc:1467] Processing CreateTablet for tablet c52f670b014a4c608807fa9b8877d481 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.483903 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52f670b014a4c608807fa9b8877d481. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.486827 12710 tablet_service.cc:1467] Processing CreateTablet for tablet c52f670b014a4c608807fa9b8877d481 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:38.487191 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52f670b014a4c608807fa9b8877d481. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:38.491011 14070 tablet_bootstrap.cc:492] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:38.492756 14070 tablet_bootstrap.cc:654] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.493494 14069 tablet_bootstrap.cc:492] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:38.494028 14068 tablet_bootstrap.cc:492] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:38.495255 14069 tablet_bootstrap.cc:654] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.495774 14068 tablet_bootstrap.cc:654] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.503381 14069 tablet_bootstrap.cc:492] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:38.503562 14069 ts_tablet_manager.cc:1367] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 08:17:38.504365 14069 raft_consensus.cc:348] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.504658 14069 raft_consensus.cc:374] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.504808 14069 raft_consensus.cc:729] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.505146 14069 consensus_queue.cc:260] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.505757 14069 ts_tablet_manager.cc:1398] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:38.506750 14069 tablet_bootstrap.cc:492] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:38.507035 14074 tablet_bootstrap.cc:492] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:38.508282 14069 tablet_bootstrap.cc:654] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.508700 14074 tablet_bootstrap.cc:654] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.509155 14068 tablet_bootstrap.cc:492] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:38.509335 14068 ts_tablet_manager.cc:1367] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20240502 08:17:38.510108 14068 raft_consensus.cc:348] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.510375 14068 raft_consensus.cc:374] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.510514 14068 raft_consensus.cc:729] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.510737 14070 tablet_bootstrap.cc:492] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:38.510758 14074 tablet_bootstrap.cc:492] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:38.510874 14070 ts_tablet_manager.cc:1367] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20240502 08:17:38.510877 14074 ts_tablet_manager.cc:1367] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:38.510807 14068 consensus_queue.cc:260] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.511384 14068 ts_tablet_manager.cc:1398] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:38.511646 14068 tablet_bootstrap.cc:492] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:38.511636 14070 raft_consensus.cc:348] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.511809 14070 raft_consensus.cc:374] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.511721 14074 raft_consensus.cc:348] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.511902 14070 raft_consensus.cc:729] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.511946 14074 raft_consensus.cc:374] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.512032 14074 raft_consensus.cc:729] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.512136 14070 consensus_queue.cc:260] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.512274 14074 consensus_queue.cc:260] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.512756 14074 ts_tablet_manager.cc:1398] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:38.513021 14068 tablet_bootstrap.cc:654] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.513424 14070 ts_tablet_manager.cc:1398] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:38.513682 14070 tablet_bootstrap.cc:492] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:38.513824 14074 tablet_bootstrap.cc:492] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:38.515074 14070 tablet_bootstrap.cc:654] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.516222 14074 tablet_bootstrap.cc:654] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.526036 14068 tablet_bootstrap.cc:492] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:38.526253 14068 ts_tablet_manager.cc:1367] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240502 08:17:38.526362 14074 tablet_bootstrap.cc:492] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:38.526546 14074 ts_tablet_manager.cc:1367] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240502 08:17:38.527235 14074 raft_consensus.cc:348] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:38.527443 14074 raft_consensus.cc:374] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.527545 14074 raft_consensus.cc:729] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.527601 14069 tablet_bootstrap.cc:492] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:38.527539 14068 raft_consensus.cc:348] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:38.527750 14069 ts_tablet_manager.cc:1367] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.007s
I20240502 08:17:38.527750 14068 raft_consensus.cc:374] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.527825 14068 raft_consensus.cc:729] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.527806 14074 consensus_queue.cc:260] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:38.528070 14068 consensus_queue.cc:260] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:38.528563 14068 ts_tablet_manager.cc:1398] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:38.528584 14070 tablet_bootstrap.cc:492] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:38.528504 14069 raft_consensus.cc:348] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.528694 14069 raft_consensus.cc:374] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.528793 14069 raft_consensus.cc:729] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.528831 14070 ts_tablet_manager.cc:1367] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20240502 08:17:38.529048 14069 consensus_queue.cc:260] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.529659 14069 ts_tablet_manager.cc:1398] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:38.529678 14070 raft_consensus.cc:348] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:38.529989 14070 raft_consensus.cc:374] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.530117 14070 raft_consensus.cc:729] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.530418 14070 consensus_queue.cc:260] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:38.531013 14070 ts_tablet_manager.cc:1398] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:38.531508 14074 ts_tablet_manager.cc:1398] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:38.532579 14074 tablet_bootstrap.cc:492] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:38.534150 14074 tablet_bootstrap.cc:654] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:38.535835 14074 tablet_bootstrap.cc:492] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:38.535944 14074 ts_tablet_manager.cc:1367] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:17:38.536438 14074 raft_consensus.cc:348] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.536605 14074 raft_consensus.cc:374] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:38.536691 14074 raft_consensus.cc:729] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:38.536886 14074 consensus_queue.cc:260] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.537297 14074 ts_tablet_manager.cc:1398] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:38.661856 14063 raft_consensus.cc:482] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:38.662012 14063 raft_consensus.cc:504] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:38.662701 14063 leader_election.cc:288] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:38.663185 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d9fdffe8fb40009ca236f998ee9dba" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:38.663179 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d9fdffe8fb40009ca236f998ee9dba" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:38.663393 12729 raft_consensus.cc:2454] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:38.663399 12862 raft_consensus.cc:2454] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:38.663794 12926 leader_election.cc:302] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:38.664032 14063 raft_consensus.cc:2781] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:38.664155 14063 raft_consensus.cc:482] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:38.664243 14063 raft_consensus.cc:3037] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:38.666025 14063 raft_consensus.cc:504] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:38.666534 14063 leader_election.cc:288] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:38.666944 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d9fdffe8fb40009ca236f998ee9dba" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:38.666987 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d9fdffe8fb40009ca236f998ee9dba" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:38.667120 12729 raft_consensus.cc:3037] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:38.667151 12862 raft_consensus.cc:3037] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:38.669256 12862 raft_consensus.cc:2454] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:38.669401 12729 raft_consensus.cc:2454] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:38.669588 12926 leader_election.cc:302] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:38.669872 14063 raft_consensus.cc:2781] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:38.670001 14063 raft_consensus.cc:686] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:38.670266 14063 consensus_queue.cc:237] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:38.672673 12585 catalog_manager.cc:5484] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:17:38.724104 14059 raft_consensus.cc:482] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:38.724296 14059 raft_consensus.cc:504] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.725072 14059 leader_election.cc:288] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:38.725481 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52f670b014a4c608807fa9b8877d481" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:38.725533 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52f670b014a4c608807fa9b8877d481" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:38.725710 12729 raft_consensus.cc:2454] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:38.725744 12992 raft_consensus.cc:2454] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:38.726096 13056 leader_election.cc:302] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:38.726337 14059 raft_consensus.cc:2781] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:38.726482 14059 raft_consensus.cc:482] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:38.726568 14059 raft_consensus.cc:3037] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:38.728574 14059 raft_consensus.cc:504] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.729055 14059 leader_election.cc:288] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:38.729427 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52f670b014a4c608807fa9b8877d481" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:38.729480 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52f670b014a4c608807fa9b8877d481" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:38.729598 12729 raft_consensus.cc:3037] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:38.729657 12992 raft_consensus.cc:3037] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:38.731503 12992 raft_consensus.cc:2454] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:38.731652 12729 raft_consensus.cc:2454] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:38.731865 13056 leader_election.cc:302] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:38.732128 14059 raft_consensus.cc:2781] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:38.732267 14059 raft_consensus.cc:686] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:38.732524 14059 consensus_queue.cc:237] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.735296 12585 catalog_manager.cc:5484] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:38.750284 14059 raft_consensus.cc:482] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:38.750440 14059 raft_consensus.cc:504] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.750964 14059 leader_election.cc:288] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:38.751281 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afea0bbd4c0d41d4a63d33b21f542038" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:38.751407 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afea0bbd4c0d41d4a63d33b21f542038" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:38.751487 12729 raft_consensus.cc:2454] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:38.751618 12862 raft_consensus.cc:2454] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:38.751828 13056 leader_election.cc:302] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:38.752068 14059 raft_consensus.cc:2781] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:38.752151 14059 raft_consensus.cc:482] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:38.752249 14059 raft_consensus.cc:3037] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:38.753932 14059 raft_consensus.cc:504] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.754508 14059 leader_election.cc:288] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:38.754763 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afea0bbd4c0d41d4a63d33b21f542038" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:38.754945 12729 raft_consensus.cc:3037] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:38.754940 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afea0bbd4c0d41d4a63d33b21f542038" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:38.755125 12862 raft_consensus.cc:3037] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:38.757613 12729 raft_consensus.cc:2454] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:38.757663 12862 raft_consensus.cc:2454] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:38.757988 13056 leader_election.cc:302] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:38.758216 14059 raft_consensus.cc:2781] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:38.758381 14059 raft_consensus.cc:686] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:38.758622 14059 consensus_queue.cc:237] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:38.761581 12585 catalog_manager.cc:5484] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
W20240502 08:17:39.098028 14084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:39.098315 14084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:39.105566 14084 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 08:17:39.228046 14059 consensus_queue.cc:1035] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:39.231348 14063 consensus_queue.cc:1035] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:39.244464 14063 consensus_queue.cc:1035] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:39.251438 14059 consensus_queue.cc:1035] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:39.307277 14124 consensus_queue.cc:1035] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:39.307440 14126 consensus_queue.cc:1035] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 284.68 ms
time per row: 0.14234 ms
I20240502 08:17:39.530990 12588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:39.539172 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa42a21c83942f789dcebed741b52ae (DEFAULT_TABLE table=rebalance_test_table_5 [id=f63b67882a8d4f5d8817dc937654d88e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:39.539644 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa42a21c83942f789dcebed741b52ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:39.539582 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa42a21c83942f789dcebed741b52ae (DEFAULT_TABLE table=rebalance_test_table_5 [id=f63b67882a8d4f5d8817dc937654d88e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:39.539757 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa42a21c83942f789dcebed741b52ae (DEFAULT_TABLE table=rebalance_test_table_5 [id=f63b67882a8d4f5d8817dc937654d88e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:39.539978 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa42a21c83942f789dcebed741b52ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:39.540155 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa42a21c83942f789dcebed741b52ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:39.541076 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 5be94bf194b04e1d965274a968f30fbf (DEFAULT_TABLE table=rebalance_test_table_5 [id=f63b67882a8d4f5d8817dc937654d88e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:39.541465 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be94bf194b04e1d965274a968f30fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:39.546844 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 5b29a9d8d26b4a388c89fb8e3a0fefe3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f63b67882a8d4f5d8817dc937654d88e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:39.547228 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b29a9d8d26b4a388c89fb8e3a0fefe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:39.551786 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 5be94bf194b04e1d965274a968f30fbf (DEFAULT_TABLE table=rebalance_test_table_5 [id=f63b67882a8d4f5d8817dc937654d88e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:39.551962 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 5be94bf194b04e1d965274a968f30fbf (DEFAULT_TABLE table=rebalance_test_table_5 [id=f63b67882a8d4f5d8817dc937654d88e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:39.552129 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be94bf194b04e1d965274a968f30fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:39.552317 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be94bf194b04e1d965274a968f30fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:39.554359 14149 tablet_bootstrap.cc:492] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:39.554771 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 5b29a9d8d26b4a388c89fb8e3a0fefe3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f63b67882a8d4f5d8817dc937654d88e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:39.555090 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b29a9d8d26b4a388c89fb8e3a0fefe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:39.555605 14150 tablet_bootstrap.cc:492] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:39.556018 14149 tablet_bootstrap.cc:654] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:39.557235 14150 tablet_bootstrap.cc:654] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:39.558516 14151 tablet_bootstrap.cc:492] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:39.558877 12710 tablet_service.cc:1467] Processing CreateTablet for tablet 5b29a9d8d26b4a388c89fb8e3a0fefe3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=f63b67882a8d4f5d8817dc937654d88e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:39.559279 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b29a9d8d26b4a388c89fb8e3a0fefe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:39.560303 14149 tablet_bootstrap.cc:492] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:39.560739 14149 ts_tablet_manager.cc:1367] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240502 08:17:39.561050 14151 tablet_bootstrap.cc:654] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:39.561548 14149 raft_consensus.cc:348] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:39.561750 14149 raft_consensus.cc:374] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:39.561842 14149 raft_consensus.cc:729] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:39.562099 14149 consensus_queue.cc:260] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:39.565209 14149 ts_tablet_manager.cc:1398] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:17:39.566054 14149 tablet_bootstrap.cc:492] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:39.566928 14150 tablet_bootstrap.cc:492] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:39.567116 14150 ts_tablet_manager.cc:1367] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240502 08:17:39.567926 14155 tablet_bootstrap.cc:492] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:39.568007 14151 tablet_bootstrap.cc:492] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:39.567921 14150 raft_consensus.cc:348] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:39.568169 14151 ts_tablet_manager.cc:1367] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 08:17:39.568202 14150 raft_consensus.cc:374] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:39.568320 14150 raft_consensus.cc:729] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:39.568661 14150 consensus_queue.cc:260] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:39.568921 14123 raft_consensus.cc:482] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:39.568997 14151 raft_consensus.cc:348] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:39.569038 14123 raft_consensus.cc:504] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:39.569178 14151 raft_consensus.cc:374] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:39.569281 14151 raft_consensus.cc:729] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:39.569299 14149 tablet_bootstrap.cc:654] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:39.569520 14151 consensus_queue.cc:260] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:39.569696 14123 leader_election.cc:288] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:39.570168 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa42a21c83942f789dcebed741b52ae" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:39.570312 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa42a21c83942f789dcebed741b52ae" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:39.570418 12861 raft_consensus.cc:2454] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:39.570526 13121 raft_consensus.cc:2454] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:39.570878 12926 leader_election.cc:302] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:39.571094 14151 ts_tablet_manager.cc:1398] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:39.571377 14151 tablet_bootstrap.cc:492] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:39.571409 14155 tablet_bootstrap.cc:654] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:39.571410 14123 raft_consensus.cc:2781] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:39.571512 14123 raft_consensus.cc:482] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:39.571600 14123 raft_consensus.cc:3037] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.573954 14123 raft_consensus.cc:504] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:39.574268 14151 tablet_bootstrap.cc:654] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:39.574959 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa42a21c83942f789dcebed741b52ae" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:39.575140 13121 raft_consensus.cc:3037] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.577929 13121 raft_consensus.cc:2454] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:39.578372 12925 leader_election.cc:302] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:39.578672 14063 raft_consensus.cc:2781] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:39.578821 14063 raft_consensus.cc:686] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:39.579056 14063 consensus_queue.cc:237] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:39.580642 14155 tablet_bootstrap.cc:492] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:39.580791 14123 leader_election.cc:288] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:39.580896 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa42a21c83942f789dcebed741b52ae" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:39.581079 12861 raft_consensus.cc:3037] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.582412 14151 tablet_bootstrap.cc:492] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:39.582607 14151 ts_tablet_manager.cc:1367] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240502 08:17:39.583400 14151 raft_consensus.cc:348] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.583603 14151 raft_consensus.cc:374] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:39.583670 14151 raft_consensus.cc:729] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:39.583961 12861 raft_consensus.cc:2454] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:39.584029 14151 consensus_queue.cc:260] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.584224 14112 raft_consensus.cc:482] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:39.584332 14112 raft_consensus.cc:504] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.585008 14155 ts_tablet_manager.cc:1367] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20240502 08:17:39.584983 14112 leader_election.cc:288] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:39.585439 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b29a9d8d26b4a388c89fb8e3a0fefe3" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:39.585886 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b29a9d8d26b4a388c89fb8e3a0fefe3" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:39.585938 14155 raft_consensus.cc:348] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.586146 14155 raft_consensus.cc:374] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:39.586244 14155 raft_consensus.cc:729] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:39.586498 14155 consensus_queue.cc:260] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
W20240502 08:17:39.586735 12795 leader_election.cc:341] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:39.586452 14151 ts_tablet_manager.cc:1398] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:39.589063 14151 tablet_bootstrap.cc:492] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:39.590622 14155 ts_tablet_manager.cc:1398] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:17:39.591570 14151 tablet_bootstrap.cc:654] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:39.592855 14150 ts_tablet_manager.cc:1398] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240502 08:17:39.593189 14150 tablet_bootstrap.cc:492] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:39.593324 12588 catalog_manager.cc:5484] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:17:39.595283 14150 tablet_bootstrap.cc:654] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:39.595468 14149 tablet_bootstrap.cc:492] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:39.595630 14149 ts_tablet_manager.cc:1367] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.019s
I20240502 08:17:39.596288 14149 raft_consensus.cc:348] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.596488 14149 raft_consensus.cc:374] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:39.596570 14149 raft_consensus.cc:729] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:39.596805 14149 consensus_queue.cc:260] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.597277 14149 ts_tablet_manager.cc:1398] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:39.597558 14150 tablet_bootstrap.cc:492] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:39.597572 14149 tablet_bootstrap.cc:492] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:39.597697 14150 ts_tablet_manager.cc:1367] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:17:39.598635 14151 tablet_bootstrap.cc:492] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:39.598799 14151 ts_tablet_manager.cc:1367] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 08:17:39.599646 14151 raft_consensus.cc:348] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.599859 14151 raft_consensus.cc:374] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:39.599987 14151 raft_consensus.cc:729] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:39.600143 14150 raft_consensus.cc:348] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.600366 14150 raft_consensus.cc:374] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:39.600458 14150 raft_consensus.cc:729] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:39.600344 14151 consensus_queue.cc:260] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.601681 14151 ts_tablet_manager.cc:1398] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:39.603112 14149 tablet_bootstrap.cc:654] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
W20240502 08:17:39.603562 12796 leader_election.cc:341] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:39.603685 12796 leader_election.cc:302] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1dd14f7ca764a1490df419f13f2804a; no voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:17:39.603955 14112 raft_consensus.cc:2726] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:39.601567 14150 consensus_queue.cc:260] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.605993 14150 ts_tablet_manager.cc:1398] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:17:39.606212 14149 tablet_bootstrap.cc:492] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:39.606384 14149 ts_tablet_manager.cc:1367] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240502 08:17:39.607074 14149 raft_consensus.cc:348] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.607342 14149 raft_consensus.cc:374] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:39.607453 14149 raft_consensus.cc:729] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:39.607681 14149 consensus_queue.cc:260] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.608170 14149 ts_tablet_manager.cc:1398] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:39.784449 14085 raft_consensus.cc:482] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:39.784607 14085 raft_consensus.cc:504] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.785284 14085 leader_election.cc:288] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:39.785642 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b29a9d8d26b4a388c89fb8e3a0fefe3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:39.785724 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b29a9d8d26b4a388c89fb8e3a0fefe3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:39.785869 13121 raft_consensus.cc:2454] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:39.785979 12861 raft_consensus.cc:2454] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:39.786243 12663 leader_election.cc:302] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:39.786598 14085 raft_consensus.cc:2781] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:39.786746 14085 raft_consensus.cc:482] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:39.786840 14085 raft_consensus.cc:3037] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.788779 14085 raft_consensus.cc:504] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.789309 14085 leader_election.cc:288] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:39.789710 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b29a9d8d26b4a388c89fb8e3a0fefe3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:39.789706 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b29a9d8d26b4a388c89fb8e3a0fefe3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:39.789875 13121 raft_consensus.cc:3037] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.789903 12861 raft_consensus.cc:3037] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.791831 13121 raft_consensus.cc:2454] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:39.791833 12861 raft_consensus.cc:2454] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:39.792178 12663 leader_election.cc:302] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:39.792433 14085 raft_consensus.cc:2781] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:39.792570 14085 raft_consensus.cc:686] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:39.792827 14085 consensus_queue.cc:237] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.795384 12588 catalog_manager.cc:5484] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:39.863687 14112 raft_consensus.cc:482] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:39.863823 14112 raft_consensus.cc:504] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.864311 14112 leader_election.cc:288] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:39.864629 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be94bf194b04e1d965274a968f30fbf" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:39.864687 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be94bf194b04e1d965274a968f30fbf" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:39.864830 13121 raft_consensus.cc:2454] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:39.864884 12992 raft_consensus.cc:2454] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:39.865134 12795 leader_election.cc:302] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:39.865422 14112 raft_consensus.cc:2781] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:39.865525 14112 raft_consensus.cc:482] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:39.865619 14112 raft_consensus.cc:3037] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.867396 14112 raft_consensus.cc:504] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.867904 14112 leader_election.cc:288] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:39.868232 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be94bf194b04e1d965274a968f30fbf" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:39.868275 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be94bf194b04e1d965274a968f30fbf" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:39.868400 13121 raft_consensus.cc:3037] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.868418 12992 raft_consensus.cc:3037] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:39.870251 13121 raft_consensus.cc:2454] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:39.870587 12992 raft_consensus.cc:2454] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:39.870644 12795 leader_election.cc:302] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:39.870911 14112 raft_consensus.cc:2781] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:39.871248 14112 raft_consensus.cc:686] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:17:39.871522 14112 consensus_queue.cc:237] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:39.873996 12583 catalog_manager.cc:5484] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:40.098199 14063 consensus_queue.cc:1035] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:40.103427 14169 consensus_queue.cc:1035] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:40.240748 14166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:40.241046 14166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:40.248198 14166 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:40.251521 14085 consensus_queue.cc:1035] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:40.257102 14016 consensus_queue.cc:1035] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240502 08:17:40.344753 14112 consensus_queue.cc:1035] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:40.359344 14112 consensus_queue.cc:1035] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 248.542 ms
time per row: 0.124271 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:41.491003 12585 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:41.499431 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef7e83665894c1cb10cd470229cc417 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.499857 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef7e83665894c1cb10cd470229cc417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.499882 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef7e83665894c1cb10cd470229cc417 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.500327 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef7e83665894c1cb10cd470229cc417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.501247 12710 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef7e83665894c1cb10cd470229cc417 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.501655 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef7e83665894c1cb10cd470229cc417. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.503626 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.504045 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.504274 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.504643 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.508543 12841 tablet_service.cc:1467] Processing CreateTablet for tablet c662cee7c7bc4465b3c3050a3e64eea6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.508710 14240 tablet_bootstrap.cc:492] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:41.508898 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c662cee7c7bc4465b3c3050a3e64eea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.508894 14239 tablet_bootstrap.cc:492] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:41.510181 12709 tablet_service.cc:1467] Processing CreateTablet for tablet c662cee7c7bc4465b3c3050a3e64eea6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.510502 14240 tablet_bootstrap.cc:654] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.510526 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c662cee7c7bc4465b3c3050a3e64eea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.510553 14239 tablet_bootstrap.cc:654] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.511193 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.511590 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.511193 12971 tablet_service.cc:1467] Processing CreateTablet for tablet c662cee7c7bc4465b3c3050a3e64eea6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:41.511881 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c662cee7c7bc4465b3c3050a3e64eea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:41.525739 14244 tablet_bootstrap.cc:492] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:41.527565 14242 tablet_bootstrap.cc:492] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:41.529731 14242 tablet_bootstrap.cc:654] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.532619 14244 tablet_bootstrap.cc:654] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.533288 14239 tablet_bootstrap.cc:492] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:41.533433 14239 ts_tablet_manager.cc:1367] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.006s
I20240502 08:17:41.533449 14242 tablet_bootstrap.cc:492] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:41.533573 14242 ts_tablet_manager.cc:1367] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 08:17:41.534102 14242 raft_consensus.cc:348] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.534302 14242 raft_consensus.cc:374] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.534198 14239 raft_consensus.cc:348] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.534391 14242 raft_consensus.cc:729] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.534396 14239 raft_consensus.cc:374] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.534505 14239 raft_consensus.cc:729] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.534612 14242 consensus_queue.cc:260] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.534732 14239 consensus_queue.cc:260] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.535105 14242 ts_tablet_manager.cc:1398] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:17:41.535362 14242 tablet_bootstrap.cc:492] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:41.535342 14244 tablet_bootstrap.cc:492] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:41.535377 14239 ts_tablet_manager.cc:1398] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:41.535490 14244 ts_tablet_manager.cc:1367] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240502 08:17:41.535609 14239 tablet_bootstrap.cc:492] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:41.534989 14240 tablet_bootstrap.cc:492] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:41.535789 14240 ts_tablet_manager.cc:1367] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.011s
I20240502 08:17:41.536173 14244 raft_consensus.cc:348] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.536399 14244 raft_consensus.cc:374] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.536485 14244 raft_consensus.cc:729] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.536697 14244 consensus_queue.cc:260] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.537212 14244 ts_tablet_manager.cc:1398] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:41.537521 14244 tablet_bootstrap.cc:492] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:41.538765 14240 raft_consensus.cc:348] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:41.538995 14240 raft_consensus.cc:374] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.539008 14242 tablet_bootstrap.cc:654] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.539125 14240 raft_consensus.cc:729] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.539394 14240 consensus_queue.cc:260] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:41.540570 14240 ts_tablet_manager.cc:1398] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:41.542655 14239 tablet_bootstrap.cc:654] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.543298 14244 tablet_bootstrap.cc:654] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.551448 14239 tablet_bootstrap.cc:492] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:41.551584 14239 ts_tablet_manager.cc:1367] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.005s
I20240502 08:17:41.551899 14244 tablet_bootstrap.cc:492] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:41.552026 14244 ts_tablet_manager.cc:1367] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20240502 08:17:41.552518 14242 tablet_bootstrap.cc:492] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:41.552500 14244 raft_consensus.cc:348] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.552641 14244 raft_consensus.cc:374] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.552650 14242 ts_tablet_manager.cc:1367] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20240502 08:17:41.552707 14244 raft_consensus.cc:729] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.552920 14244 consensus_queue.cc:260] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.553299 14244 ts_tablet_manager.cc:1398] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:41.553589 14242 raft_consensus.cc:348] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:41.553859 14239 raft_consensus.cc:348] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:41.554114 14242 raft_consensus.cc:374] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.554208 14242 raft_consensus.cc:729] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.554484 14242 consensus_queue.cc:260] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:41.555013 14242 ts_tablet_manager.cc:1398] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:41.555297 14242 tablet_bootstrap.cc:492] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:41.555429 14239 raft_consensus.cc:374] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.555516 14239 raft_consensus.cc:729] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.555745 14239 consensus_queue.cc:260] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:41.556245 14239 ts_tablet_manager.cc:1398] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:17:41.556540 14239 tablet_bootstrap.cc:492] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:41.557935 14239 tablet_bootstrap.cc:654] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.559921 14242 tablet_bootstrap.cc:654] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:41.561364 14242 tablet_bootstrap.cc:492] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:41.561501 14242 ts_tablet_manager.cc:1367] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240502 08:17:41.561965 14242 raft_consensus.cc:348] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.562109 14242 raft_consensus.cc:374] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.562193 14242 raft_consensus.cc:729] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.562379 14242 consensus_queue.cc:260] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.562904 14242 ts_tablet_manager.cc:1398] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:41.565905 14239 tablet_bootstrap.cc:492] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:41.566010 14239 ts_tablet_manager.cc:1367] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240502 08:17:41.566565 14239 raft_consensus.cc:348] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.566723 14239 raft_consensus.cc:374] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:41.566816 14239 raft_consensus.cc:729] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:41.566989 14239 consensus_queue.cc:260] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.567361 14239 ts_tablet_manager.cc:1398] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:41.630177 14169 raft_consensus.cc:482] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:41.630360 14169 raft_consensus.cc:504] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.631114 14169 leader_election.cc:288] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:41.631505 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c662cee7c7bc4465b3c3050a3e64eea6" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:41.631575 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c662cee7c7bc4465b3c3050a3e64eea6" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:41.631743 12860 raft_consensus.cc:2454] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:41.631781 12729 raft_consensus.cc:2454] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:41.632128 12926 leader_election.cc:302] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:41.632385 14169 raft_consensus.cc:2781] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:41.632472 14169 raft_consensus.cc:482] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:41.632565 14169 raft_consensus.cc:3037] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.635221 14169 raft_consensus.cc:504] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.635747 14169 leader_election.cc:288] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:41.636123 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c662cee7c7bc4465b3c3050a3e64eea6" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:41.636178 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c662cee7c7bc4465b3c3050a3e64eea6" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:41.636327 12860 raft_consensus.cc:3037] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.636384 12729 raft_consensus.cc:3037] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.638094 12860 raft_consensus.cc:2454] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:41.638414 12729 raft_consensus.cc:2454] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:41.638484 12926 leader_election.cc:302] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:41.638753 14169 raft_consensus.cc:2781] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:41.639159 14169 raft_consensus.cc:686] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:41.639438 14169 consensus_queue.cc:237] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.641942 12586 catalog_manager.cc:5484] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:41.660362 14016 raft_consensus.cc:482] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:41.660487 14016 raft_consensus.cc:504] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:41.661052 14016 leader_election.cc:288] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:41.661409 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1eb8c0e302416bbae1e7cd1e4a86ce" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:41.661468 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1eb8c0e302416bbae1e7cd1e4a86ce" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:41.661592 12992 raft_consensus.cc:2454] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:41.661698 13120 raft_consensus.cc:2454] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:41.661999 12664 leader_election.cc:302] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:41.662317 14016 raft_consensus.cc:2781] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:41.662411 14016 raft_consensus.cc:482] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:41.662528 14016 raft_consensus.cc:3037] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.664384 14016 raft_consensus.cc:504] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:41.665023 14016 leader_election.cc:288] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:41.665238 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1eb8c0e302416bbae1e7cd1e4a86ce" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:41.665419 12992 raft_consensus.cc:3037] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.665475 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a1eb8c0e302416bbae1e7cd1e4a86ce" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:41.665681 13120 raft_consensus.cc:3037] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.668015 12992 raft_consensus.cc:2454] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:41.668197 13120 raft_consensus.cc:2454] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:41.668403 12664 leader_election.cc:302] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:41.668675 14016 raft_consensus.cc:2781] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:41.669013 14016 raft_consensus.cc:686] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:41.669258 14016 consensus_queue.cc:237] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:41.671869 12586 catalog_manager.cc:5484] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:41.888968 14112 raft_consensus.cc:482] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:41.889155 14112 raft_consensus.cc:504] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.889737 14112 leader_election.cc:288] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:41.890141 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef7e83665894c1cb10cd470229cc417" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:41.890240 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef7e83665894c1cb10cd470229cc417" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:41.890388 12992 raft_consensus.cc:2454] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:41.890453 12729 raft_consensus.cc:2454] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:41.890837 12796 leader_election.cc:302] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:41.891108 14112 raft_consensus.cc:2781] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:41.891216 14112 raft_consensus.cc:482] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:41.891319 14112 raft_consensus.cc:3037] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.893194 14112 raft_consensus.cc:504] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.893714 14112 leader_election.cc:288] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:41.894076 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef7e83665894c1cb10cd470229cc417" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:41.894148 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef7e83665894c1cb10cd470229cc417" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:41.894276 12992 raft_consensus.cc:3037] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.894330 12729 raft_consensus.cc:3037] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:41.896960 12729 raft_consensus.cc:2454] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:41.896956 12992 raft_consensus.cc:2454] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:41.897512 12796 leader_election.cc:302] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:41.897765 14112 raft_consensus.cc:2781] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:41.898206 14112 raft_consensus.cc:686] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:17:41.898602 14112 consensus_queue.cc:237] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:41.901887 12586 catalog_manager.cc:5484] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:42.043686 14169 consensus_queue.cc:1035] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:42.055807 14169 consensus_queue.cc:1035] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:42.249421 14254 consensus_queue.cc:1035] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:42.253954 14016 consensus_queue.cc:1035] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:42.299502 14259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:42.299785 14259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:42.306908 14259 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 08:17:42.386169 14256 consensus_queue.cc:1035] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:42.396224 14112 consensus_queue.cc:1035] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 236.577 ms
time per row: 0.118289 ms
I20240502 08:17:42.685479 12586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:42.693967 12840 tablet_service.cc:1467] Processing CreateTablet for tablet f031f5fedb904eaf83a4902092280295 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ed6dd5b8e8d45edb609b77e58525533]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.694412 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f031f5fedb904eaf83a4902092280295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.694696 12709 tablet_service.cc:1467] Processing CreateTablet for tablet f031f5fedb904eaf83a4902092280295 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ed6dd5b8e8d45edb609b77e58525533]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.695133 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f031f5fedb904eaf83a4902092280295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.695091 12971 tablet_service.cc:1467] Processing CreateTablet for tablet c63cca9cd28f4d30bf901e1de1cd6461 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ed6dd5b8e8d45edb609b77e58525533]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.695515 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63cca9cd28f4d30bf901e1de1cd6461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.695452 13100 tablet_service.cc:1467] Processing CreateTablet for tablet c63cca9cd28f4d30bf901e1de1cd6461 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ed6dd5b8e8d45edb609b77e58525533]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.695817 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63cca9cd28f4d30bf901e1de1cd6461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.698908 13101 tablet_service.cc:1467] Processing CreateTablet for tablet f031f5fedb904eaf83a4902092280295 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ed6dd5b8e8d45edb609b77e58525533]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.699266 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f031f5fedb904eaf83a4902092280295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.701335 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 9366b55ca7dc48bfa5d2ed650d7a30e8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ed6dd5b8e8d45edb609b77e58525533]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.701681 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9366b55ca7dc48bfa5d2ed650d7a30e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.703332 12841 tablet_service.cc:1467] Processing CreateTablet for tablet c63cca9cd28f4d30bf901e1de1cd6461 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ed6dd5b8e8d45edb609b77e58525533]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.703699 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63cca9cd28f4d30bf901e1de1cd6461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.707490 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 9366b55ca7dc48bfa5d2ed650d7a30e8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ed6dd5b8e8d45edb609b77e58525533]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.707664 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 9366b55ca7dc48bfa5d2ed650d7a30e8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7ed6dd5b8e8d45edb609b77e58525533]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:42.707871 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9366b55ca7dc48bfa5d2ed650d7a30e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.708029 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9366b55ca7dc48bfa5d2ed650d7a30e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:42.712038 14327 tablet_bootstrap.cc:492] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:42.715667 14327 tablet_bootstrap.cc:654] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.717185 14329 tablet_bootstrap.cc:492] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:42.719105 14329 tablet_bootstrap.cc:654] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.719338 14328 tablet_bootstrap.cc:492] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:42.720409 14327 tablet_bootstrap.cc:492] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:42.720724 14326 tablet_bootstrap.cc:492] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:42.720939 14327 ts_tablet_manager.cc:1367] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 08:17:42.720978 14328 tablet_bootstrap.cc:654] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.721788 14327 raft_consensus.cc:348] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:42.722086 14327 raft_consensus.cc:374] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.722220 14327 raft_consensus.cc:729] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.722635 14326 tablet_bootstrap.cc:654] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.722561 14327 consensus_queue.cc:260] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:42.723335 14327 ts_tablet_manager.cc:1398] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:42.723680 14327 tablet_bootstrap.cc:492] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:42.725258 14327 tablet_bootstrap.cc:654] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.729111 14329 tablet_bootstrap.cc:492] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:42.729321 14329 ts_tablet_manager.cc:1367] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 08:17:42.730162 14329 raft_consensus.cc:348] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:42.730450 14329 raft_consensus.cc:374] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.730609 14329 raft_consensus.cc:729] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.730908 14329 consensus_queue.cc:260] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:42.733958 14328 tablet_bootstrap.cc:492] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:42.734066 14329 ts_tablet_manager.cc:1398] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:17:42.734184 14328 ts_tablet_manager.cc:1367] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20240502 08:17:42.735666 14328 raft_consensus.cc:348] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:42.736325 14328 raft_consensus.cc:374] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.736464 14328 raft_consensus.cc:729] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.736711 14329 tablet_bootstrap.cc:492] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:42.736742 14328 consensus_queue.cc:260] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:42.737548 14328 ts_tablet_manager.cc:1398] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:42.737855 14328 tablet_bootstrap.cc:492] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:42.738905 14329 tablet_bootstrap.cc:654] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.741673 14326 tablet_bootstrap.cc:492] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:42.741853 14326 ts_tablet_manager.cc:1367] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.010s
I20240502 08:17:42.741950 14329 tablet_bootstrap.cc:492] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:42.742142 14329 ts_tablet_manager.cc:1367] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 08:17:42.742710 14328 tablet_bootstrap.cc:654] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.742676 14326 raft_consensus.cc:348] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:42.742929 14326 raft_consensus.cc:374] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.742892 14329 raft_consensus.cc:348] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:42.743046 14326 raft_consensus.cc:729] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.743136 14329 raft_consensus.cc:374] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.743256 14329 raft_consensus.cc:729] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.743394 14326 consensus_queue.cc:260] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:42.743592 14329 consensus_queue.cc:260] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:42.744879 14326 ts_tablet_manager.cc:1398] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:42.745072 14329 ts_tablet_manager.cc:1398] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:42.745193 14326 tablet_bootstrap.cc:492] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:42.746789 14326 tablet_bootstrap.cc:654] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.747987 14167 raft_consensus.cc:482] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:42.748121 14167 raft_consensus.cc:504] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:42.748770 14167 leader_election.cc:288] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:42.749194 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f031f5fedb904eaf83a4902092280295" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:42.749428 12729 raft_consensus.cc:2454] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:42.749796 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f031f5fedb904eaf83a4902092280295" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:42.749985 12861 raft_consensus.cc:2454] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:42.750496 13056 leader_election.cc:302] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:42.750939 14167 raft_consensus.cc:2781] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:42.751046 14167 raft_consensus.cc:482] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:42.751133 14167 raft_consensus.cc:3037] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.752175 14327 tablet_bootstrap.cc:492] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:42.752254 14326 tablet_bootstrap.cc:492] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:42.752338 14327 ts_tablet_manager.cc:1367] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.020s
I20240502 08:17:42.752379 14326 ts_tablet_manager.cc:1367] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20240502 08:17:42.753048 14326 raft_consensus.cc:348] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:42.753049 14327 raft_consensus.cc:348] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:42.753232 14326 raft_consensus.cc:374] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.753237 14327 raft_consensus.cc:374] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.753296 14326 raft_consensus.cc:729] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.753309 14327 raft_consensus.cc:729] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.753523 14327 consensus_queue.cc:260] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:42.753564 14326 consensus_queue.cc:260] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:42.754030 14327 ts_tablet_manager.cc:1398] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:42.755030 14326 ts_tablet_manager.cc:1398] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:42.755254 14167 raft_consensus.cc:504] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:42.756310 14328 tablet_bootstrap.cc:492] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:42.756465 14328 ts_tablet_manager.cc:1367] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.001s
I20240502 08:17:42.756706 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f031f5fedb904eaf83a4902092280295" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:42.756896 12861 raft_consensus.cc:3037] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.757089 14328 raft_consensus.cc:348] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:42.757248 14328 raft_consensus.cc:374] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.757294 14328 raft_consensus.cc:729] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.757476 14328 consensus_queue.cc:260] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:42.757887 14328 ts_tablet_manager.cc:1398] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:42.758167 14328 tablet_bootstrap.cc:492] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:42.759644 12861 raft_consensus.cc:2454] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:42.759784 14328 tablet_bootstrap.cc:654] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:42.761456 14167 leader_election.cc:288] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:42.761782 13056 leader_election.cc:302] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:42.762082 14167 raft_consensus.cc:2781] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:42.762333 14167 raft_consensus.cc:686] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:42.762281 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f031f5fedb904eaf83a4902092280295" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:42.762539 12729 raft_consensus.cc:3037] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.762769 14167 consensus_queue.cc:237] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:42.763777 14328 tablet_bootstrap.cc:492] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:42.763958 14328 ts_tablet_manager.cc:1367] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:17:42.764756 14328 raft_consensus.cc:348] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:42.764945 14328 raft_consensus.cc:374] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:42.764971 12729 raft_consensus.cc:2454] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:42.765041 14328 raft_consensus.cc:729] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:42.765236 14328 consensus_queue.cc:260] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:42.765736 14328 ts_tablet_manager.cc:1398] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:42.765779 12586 catalog_manager.cc:5484] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:17:42.808868 14263 raft_consensus.cc:482] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:42.808995 14263 raft_consensus.cc:504] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:42.809583 14263 leader_election.cc:288] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:42.809909 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63cca9cd28f4d30bf901e1de1cd6461" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:42.809934 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63cca9cd28f4d30bf901e1de1cd6461" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:42.810087 13120 raft_consensus.cc:2454] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:42.810101 12861 raft_consensus.cc:2454] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:42.810474 12926 leader_election.cc:302] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:42.810808 14263 raft_consensus.cc:2781] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:42.810901 14263 raft_consensus.cc:482] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:42.810977 14263 raft_consensus.cc:3037] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.812747 14263 raft_consensus.cc:504] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:42.813237 14263 leader_election.cc:288] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:42.813612 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63cca9cd28f4d30bf901e1de1cd6461" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:42.813659 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63cca9cd28f4d30bf901e1de1cd6461" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:42.813799 13120 raft_consensus.cc:3037] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.813815 12861 raft_consensus.cc:3037] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:42.816416 12861 raft_consensus.cc:2454] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:42.816428 13120 raft_consensus.cc:2454] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:42.816747 12925 leader_election.cc:302] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:42.817008 14263 raft_consensus.cc:2781] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:42.817126 14263 raft_consensus.cc:686] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:42.817371 14263 consensus_queue.cc:237] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:42.819797 12586 catalog_manager.cc:5484] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:43.163545 14254 raft_consensus.cc:482] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:43.163697 14254 raft_consensus.cc:504] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:43.164301 14254 leader_election.cc:288] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:43.164638 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9366b55ca7dc48bfa5d2ed650d7a30e8" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:43.164739 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9366b55ca7dc48bfa5d2ed650d7a30e8" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:43.164916 12992 raft_consensus.cc:2454] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:43.164911 13120 raft_consensus.cc:2454] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:43.165280 12663 leader_election.cc:302] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:43.165555 14254 raft_consensus.cc:2781] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:43.165647 14254 raft_consensus.cc:482] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:43.165745 14254 raft_consensus.cc:3037] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:43.167584 14254 raft_consensus.cc:504] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:43.168089 14254 leader_election.cc:288] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:43.168385 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9366b55ca7dc48bfa5d2ed650d7a30e8" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:43.168452 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9366b55ca7dc48bfa5d2ed650d7a30e8" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:43.168552 13120 raft_consensus.cc:3037] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:43.168633 12992 raft_consensus.cc:3037] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:43.170362 13120 raft_consensus.cc:2454] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:43.170411 12992 raft_consensus.cc:2454] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:43.170753 12663 leader_election.cc:302] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:43.171013 14254 raft_consensus.cc:2781] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:43.171311 14254 raft_consensus.cc:686] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:43.171561 14254 consensus_queue.cc:237] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:43.174281 12586 catalog_manager.cc:5484] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:17:43.256003 14263 consensus_queue.cc:1035] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:43.260495 14263 consensus_queue.cc:1035] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:43.304109 14167 consensus_queue.cc:1035] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:43.308759 14167 consensus_queue.cc:1035] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:43.582753 14348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:43.583009 14348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:43.590003 14348 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 08:17:43.710382 14254 consensus_queue.cc:1035] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:43.716948 14254 consensus_queue.cc:1035] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 199.055 ms
time per row: 0.0995277 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:44.798880 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:44.807976 12840 tablet_service.cc:1467] Processing CreateTablet for tablet bf1c03ab6bf840bf8fc70311bad73440 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.808364 12709 tablet_service.cc:1467] Processing CreateTablet for tablet bf1c03ab6bf840bf8fc70311bad73440 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.808347 13101 tablet_service.cc:1467] Processing CreateTablet for tablet bf1c03ab6bf840bf8fc70311bad73440 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.808450 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1c03ab6bf840bf8fc70311bad73440. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.808740 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1c03ab6bf840bf8fc70311bad73440. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.808754 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1c03ab6bf840bf8fc70311bad73440. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.810529 13102 tablet_service.cc:1467] Processing CreateTablet for tablet ecfefc1736ae4d5caade9e00af8a3718 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.810886 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecfefc1736ae4d5caade9e00af8a3718. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.815474 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd28596a64c49139e2b27e2dc68ad68 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.815618 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd28596a64c49139e2b27e2dc68ad68 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.815896 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd28596a64c49139e2b27e2dc68ad68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.816010 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd28596a64c49139e2b27e2dc68ad68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.818971 12971 tablet_service.cc:1467] Processing CreateTablet for tablet ecfefc1736ae4d5caade9e00af8a3718 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.819343 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecfefc1736ae4d5caade9e00af8a3718. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.822968 14407 tablet_bootstrap.cc:492] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:44.823992 14409 tablet_bootstrap.cc:492] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:44.824533 12841 tablet_service.cc:1467] Processing CreateTablet for tablet ecfefc1736ae4d5caade9e00af8a3718 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.824807 14407 tablet_bootstrap.cc:654] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.824908 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecfefc1736ae4d5caade9e00af8a3718. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.826043 14409 tablet_bootstrap.cc:654] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.835827 14409 tablet_bootstrap.cc:492] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:44.836022 14409 ts_tablet_manager.cc:1367] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20240502 08:17:44.836856 14409 raft_consensus.cc:348] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:44.838409 14409 raft_consensus.cc:374] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.838591 14409 raft_consensus.cc:729] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.838861 14409 consensus_queue.cc:260] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:44.839511 14409 ts_tablet_manager.cc:1398] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:44.840109 14408 tablet_bootstrap.cc:492] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:44.842972 14407 tablet_bootstrap.cc:492] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:44.843161 14407 ts_tablet_manager.cc:1367] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.005s
I20240502 08:17:44.843914 14407 raft_consensus.cc:348] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:44.844169 14407 raft_consensus.cc:374] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.844298 14407 raft_consensus.cc:729] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.844600 14407 consensus_queue.cc:260] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:44.845573 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd28596a64c49139e2b27e2dc68ad68 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:44.847294 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd28596a64c49139e2b27e2dc68ad68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:44.848258 14407 ts_tablet_manager.cc:1398] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:17:44.849175 14407 tablet_bootstrap.cc:492] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:44.850821 14407 tablet_bootstrap.cc:654] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.850848 14409 tablet_bootstrap.cc:492] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:44.852267 14409 tablet_bootstrap.cc:654] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.855091 14407 tablet_bootstrap.cc:492] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:44.855288 14407 ts_tablet_manager.cc:1367] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 08:17:44.856079 14408 tablet_bootstrap.cc:654] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.856027 14407 raft_consensus.cc:348] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:44.856321 14407 raft_consensus.cc:374] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.856441 14407 raft_consensus.cc:729] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.856707 14407 consensus_queue.cc:260] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:44.857977 14407 ts_tablet_manager.cc:1398] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:44.858285 14407 tablet_bootstrap.cc:492] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:44.859252 14410 tablet_bootstrap.cc:492] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:44.859848 14407 tablet_bootstrap.cc:654] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.861068 14410 tablet_bootstrap.cc:654] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.862336 14409 tablet_bootstrap.cc:492] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:44.862571 14409 ts_tablet_manager.cc:1367] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20240502 08:17:44.863265 14409 raft_consensus.cc:348] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:44.863438 14408 tablet_bootstrap.cc:492] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:44.863499 14409 raft_consensus.cc:374] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.863564 14408 ts_tablet_manager.cc:1367] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20240502 08:17:44.863564 14409 raft_consensus.cc:729] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.863792 14409 consensus_queue.cc:260] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:44.864303 14409 ts_tablet_manager.cc:1398] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:44.864318 14408 raft_consensus.cc:348] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:44.864490 14408 raft_consensus.cc:374] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.864578 14408 raft_consensus.cc:729] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.864776 14408 consensus_queue.cc:260] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:44.865253 14408 ts_tablet_manager.cc:1398] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:44.865530 14408 tablet_bootstrap.cc:492] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:44.867101 14408 tablet_bootstrap.cc:654] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.867544 14407 tablet_bootstrap.cc:492] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:44.867702 14407 ts_tablet_manager.cc:1367] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.006s
I20240502 08:17:44.868393 14407 raft_consensus.cc:348] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:44.868647 14407 raft_consensus.cc:374] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.868783 14407 raft_consensus.cc:729] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.869035 14407 consensus_queue.cc:260] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:44.872368 14407 ts_tablet_manager.cc:1398] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:17:44.874490 14410 tablet_bootstrap.cc:492] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:44.874639 14410 ts_tablet_manager.cc:1367] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240502 08:17:44.875398 14410 raft_consensus.cc:348] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:44.875662 14410 raft_consensus.cc:374] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.875751 14410 raft_consensus.cc:729] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.875974 14410 consensus_queue.cc:260] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:44.876350 14408 tablet_bootstrap.cc:492] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:44.876505 14408 ts_tablet_manager.cc:1367] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240502 08:17:44.876878 14410 ts_tablet_manager.cc:1398] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:44.877336 14410 tablet_bootstrap.cc:492] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:44.877324 14408 raft_consensus.cc:348] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:44.877530 14408 raft_consensus.cc:374] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.877609 14408 raft_consensus.cc:729] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.877828 14408 consensus_queue.cc:260] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:44.878408 14408 ts_tablet_manager.cc:1398] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:44.878824 14410 tablet_bootstrap.cc:654] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:44.881364 14410 tablet_bootstrap.cc:492] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:44.881507 14410 ts_tablet_manager.cc:1367] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:44.882133 14410 raft_consensus.cc:348] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:44.882282 14410 raft_consensus.cc:374] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:44.882347 14410 raft_consensus.cc:729] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:44.882586 14410 consensus_queue.cc:260] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:44.883038 14410 ts_tablet_manager.cc:1398] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 08:17:44.963090 14350 raft_consensus.cc:482] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:44.963253 14350 raft_consensus.cc:504] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:44.963871 14350 leader_election.cc:288] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:44.964218 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1c03ab6bf840bf8fc70311bad73440" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:44.964351 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1c03ab6bf840bf8fc70311bad73440" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:44.964434 12860 raft_consensus.cc:2454] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:44.964602 12730 raft_consensus.cc:2454] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:44.964834 13056 leader_election.cc:302] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:44.965098 14350 raft_consensus.cc:2781] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:44.965227 14350 raft_consensus.cc:482] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:44.965342 14350 raft_consensus.cc:3037] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:44.967218 14350 raft_consensus.cc:504] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:44.967761 14350 leader_election.cc:288] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:44.968039 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1c03ab6bf840bf8fc70311bad73440" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:44.968211 12860 raft_consensus.cc:3037] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:44.968210 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1c03ab6bf840bf8fc70311bad73440" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:44.968400 12730 raft_consensus.cc:3037] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:44.969985 12860 raft_consensus.cc:2454] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:44.970357 13056 leader_election.cc:302] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:44.970525 12730 raft_consensus.cc:2454] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:44.970666 14350 raft_consensus.cc:2781] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:44.971081 14350 raft_consensus.cc:686] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:44.971377 14350 consensus_queue.cc:237] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:44.973835 12580 catalog_manager.cc:5484] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:45.022716 14257 raft_consensus.cc:482] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:45.022857 14257 raft_consensus.cc:504] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:45.023437 14257 leader_election.cc:288] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:45.023878 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd28596a64c49139e2b27e2dc68ad68" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:45.023898 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd28596a64c49139e2b27e2dc68ad68" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:45.024112 12992 raft_consensus.cc:2454] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:45.024130 12730 raft_consensus.cc:2454] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:45.024510 12796 leader_election.cc:302] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:45.024762 14257 raft_consensus.cc:2781] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:45.024875 14257 raft_consensus.cc:482] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:45.024960 14257 raft_consensus.cc:3037] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.026684 14257 raft_consensus.cc:504] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:45.027151 14257 leader_election.cc:288] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:45.027632 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd28596a64c49139e2b27e2dc68ad68" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:45.027580 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd28596a64c49139e2b27e2dc68ad68" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:45.027776 12992 raft_consensus.cc:3037] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.027827 12730 raft_consensus.cc:3037] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.029724 12992 raft_consensus.cc:2454] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:45.029982 12730 raft_consensus.cc:2454] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:45.030072 12796 leader_election.cc:302] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:45.030443 14257 raft_consensus.cc:2781] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:45.030784 14257 raft_consensus.cc:686] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:17:45.031041 14257 consensus_queue.cc:237] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:45.033141 14401 raft_consensus.cc:482] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:45.033277 14401 raft_consensus.cc:504] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:45.033531 12580 catalog_manager.cc:5484] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:17:45.033854 14401 leader_election.cc:288] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:45.034183 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfefc1736ae4d5caade9e00af8a3718" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:45.034401 12860 raft_consensus.cc:2454] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:45.034413 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfefc1736ae4d5caade9e00af8a3718" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:45.034653 13121 raft_consensus.cc:2454] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:45.035076 12925 leader_election.cc:302] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:45.035332 14401 raft_consensus.cc:2781] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:45.035455 14401 raft_consensus.cc:482] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:45.035523 14401 raft_consensus.cc:3037] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.037578 14401 raft_consensus.cc:504] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:45.038012 14401 leader_election.cc:288] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:45.038479 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfefc1736ae4d5caade9e00af8a3718" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:45.038439 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecfefc1736ae4d5caade9e00af8a3718" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:45.038619 13121 raft_consensus.cc:3037] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.038636 12860 raft_consensus.cc:3037] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:45.040333 13121 raft_consensus.cc:2454] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:45.040406 12860 raft_consensus.cc:2454] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:45.040793 12925 leader_election.cc:302] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:45.041067 14401 raft_consensus.cc:2781] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:45.041193 14401 raft_consensus.cc:686] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:45.041476 14401 consensus_queue.cc:237] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:45.044268 12580 catalog_manager.cc:5484] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
W20240502 08:17:45.379606 14426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:45.379877 14426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:45.387182 14426 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 08:17:45.476040 14350 consensus_queue.cc:1035] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:45.489665 14450 consensus_queue.cc:1035] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:45.526588 14257 consensus_queue.cc:1035] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:45.534806 12730 raft_consensus.cc:1261] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:45.535424 14257 consensus_queue.cc:1035] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:45.576472 14455 mvcc.cc:205] Tried to move back new op lower bound from 7023156697225531392 to 7023156695169085440. Current Snapshot: MvccSnapshot[applied={T|T < 7023156697225531392}]
I20240502 08:17:45.577618 13121 raft_consensus.cc:1261] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:45.577916 12861 raft_consensus.cc:1261] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:45.578578 14401 consensus_queue.cc:1035] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:45.579141 14401 consensus_queue.cc:1035] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 228.853 ms
time per row: 0.114426 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:46.637099 12585 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:46.648669 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 5c3f0742355442af8a29b971921b7d45 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ab2e0dad5c0341efa8c2321070416a08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.649049 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 5c3f0742355442af8a29b971921b7d45 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ab2e0dad5c0341efa8c2321070416a08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.649200 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c3f0742355442af8a29b971921b7d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.649525 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c3f0742355442af8a29b971921b7d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.649679 12840 tablet_service.cc:1467] Processing CreateTablet for tablet ab808742ce8746ae8e9b8fbff435e7be (DEFAULT_TABLE table=rebalance_test_table_9 [id=ab2e0dad5c0341efa8c2321070416a08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.650084 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab808742ce8746ae8e9b8fbff435e7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.650044 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 5c3f0742355442af8a29b971921b7d45 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ab2e0dad5c0341efa8c2321070416a08]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.650429 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c3f0742355442af8a29b971921b7d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.654920 12971 tablet_service.cc:1467] Processing CreateTablet for tablet db051ed0438742319f741a565834b644 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ab2e0dad5c0341efa8c2321070416a08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.654999 12709 tablet_service.cc:1467] Processing CreateTablet for tablet db051ed0438742319f741a565834b644 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ab2e0dad5c0341efa8c2321070416a08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.655328 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db051ed0438742319f741a565834b644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.655328 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db051ed0438742319f741a565834b644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.658874 12710 tablet_service.cc:1467] Processing CreateTablet for tablet ab808742ce8746ae8e9b8fbff435e7be (DEFAULT_TABLE table=rebalance_test_table_9 [id=ab2e0dad5c0341efa8c2321070416a08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.659258 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab808742ce8746ae8e9b8fbff435e7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.663053 14494 tablet_bootstrap.cc:492] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:46.663219 13101 tablet_service.cc:1467] Processing CreateTablet for tablet db051ed0438742319f741a565834b644 (DEFAULT_TABLE table=rebalance_test_table_9 [id=ab2e0dad5c0341efa8c2321070416a08]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.663616 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db051ed0438742319f741a565834b644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.664736 14494 tablet_bootstrap.cc:654] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.666862 13100 tablet_service.cc:1467] Processing CreateTablet for tablet ab808742ce8746ae8e9b8fbff435e7be (DEFAULT_TABLE table=rebalance_test_table_9 [id=ab2e0dad5c0341efa8c2321070416a08]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:46.667299 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab808742ce8746ae8e9b8fbff435e7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:46.669989 14495 tablet_bootstrap.cc:492] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:46.671630 14495 tablet_bootstrap.cc:654] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.670240 14494 tablet_bootstrap.cc:492] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:46.684638 14494 ts_tablet_manager.cc:1367] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240502 08:17:46.685920 14494 raft_consensus.cc:348] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.688194 14494 raft_consensus.cc:374] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.689244 14494 raft_consensus.cc:729] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.689635 14494 consensus_queue.cc:260] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.690306 14494 ts_tablet_manager.cc:1398] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:46.698990 14494 tablet_bootstrap.cc:492] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:46.702613 14494 tablet_bootstrap.cc:654] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.709187 14492 tablet_bootstrap.cc:492] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:46.709961 14493 tablet_bootstrap.cc:492] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:46.711156 14492 tablet_bootstrap.cc:654] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.711292 14494 tablet_bootstrap.cc:492] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:46.711513 14494 ts_tablet_manager.cc:1367] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240502 08:17:46.711746 14493 tablet_bootstrap.cc:654] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.713099 14494 raft_consensus.cc:348] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:46.713336 14494 raft_consensus.cc:374] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.713454 14494 raft_consensus.cc:729] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.713725 14495 tablet_bootstrap.cc:492] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:46.713721 14494 consensus_queue.cc:260] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:46.713896 14495 ts_tablet_manager.cc:1367] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.044s user 0.020s sys 0.013s
I20240502 08:17:46.714704 14495 raft_consensus.cc:348] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.714941 14495 raft_consensus.cc:374] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.715059 14495 raft_consensus.cc:729] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.715297 14495 consensus_queue.cc:260] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.715831 14495 ts_tablet_manager.cc:1398] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:46.716785 14495 tablet_bootstrap.cc:492] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:46.718277 14495 tablet_bootstrap.cc:654] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.718959 14494 ts_tablet_manager.cc:1398] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 08:17:46.720146 14492 tablet_bootstrap.cc:492] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:46.720342 14492 ts_tablet_manager.cc:1367] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 08:17:46.721068 14492 raft_consensus.cc:348] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:46.721355 14492 raft_consensus.cc:374] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.721482 14492 raft_consensus.cc:729] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.721762 14492 consensus_queue.cc:260] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:46.722441 14492 ts_tablet_manager.cc:1398] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:46.722503 14494 tablet_bootstrap.cc:492] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:46.724030 14494 tablet_bootstrap.cc:654] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.726665 14493 tablet_bootstrap.cc:492] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:46.726864 14493 ts_tablet_manager.cc:1367] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240502 08:17:46.727530 14493 raft_consensus.cc:348] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.727790 14493 raft_consensus.cc:374] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.727926 14493 raft_consensus.cc:729] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.728127 14493 consensus_queue.cc:260] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.729210 14493 ts_tablet_manager.cc:1398] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:46.729434 14493 tablet_bootstrap.cc:492] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:46.731199 14493 tablet_bootstrap.cc:654] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.732682 14493 tablet_bootstrap.cc:492] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:46.732846 14493 ts_tablet_manager.cc:1367] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 08:17:46.733343 14493 raft_consensus.cc:348] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:46.733493 14493 raft_consensus.cc:374] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.733562 14493 raft_consensus.cc:729] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.733762 14493 consensus_queue.cc:260] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:46.734162 14493 ts_tablet_manager.cc:1398] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:46.734609 14493 tablet_bootstrap.cc:492] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:46.734987 14495 tablet_bootstrap.cc:492] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:46.735180 14495 ts_tablet_manager.cc:1367] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.008s
I20240502 08:17:46.735874 14493 tablet_bootstrap.cc:654] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:46.735877 14495 raft_consensus.cc:348] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.736182 14495 raft_consensus.cc:374] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.736286 14495 raft_consensus.cc:729] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.736531 14495 consensus_queue.cc:260] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.737139 14495 ts_tablet_manager.cc:1398] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:46.738786 14494 tablet_bootstrap.cc:492] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:46.738912 14494 ts_tablet_manager.cc:1367] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.010s
I20240502 08:17:46.739574 14494 raft_consensus.cc:348] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.739719 14494 raft_consensus.cc:374] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.739837 14494 raft_consensus.cc:729] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.740057 14494 consensus_queue.cc:260] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.741312 14494 ts_tablet_manager.cc:1398] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:46.741415 14493 tablet_bootstrap.cc:492] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:46.741545 14493 ts_tablet_manager.cc:1367] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240502 08:17:46.742902 14493 raft_consensus.cc:348] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.743031 14493 raft_consensus.cc:374] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:46.743093 14493 raft_consensus.cc:729] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:46.743275 14493 consensus_queue.cc:260] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.744127 14493 ts_tablet_manager.cc:1398] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:46.748294 14397 raft_consensus.cc:482] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:46.748404 14397 raft_consensus.cc:504] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:46.748886 14397 leader_election.cc:288] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:46.749217 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab808742ce8746ae8e9b8fbff435e7be" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:46.749373 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab808742ce8746ae8e9b8fbff435e7be" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:46.749419 12860 raft_consensus.cc:2454] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:46.749548 13120 raft_consensus.cc:2454] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:46.749812 12664 leader_election.cc:302] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:46.750061 14397 raft_consensus.cc:2781] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:46.750166 14397 raft_consensus.cc:482] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:46.750232 14397 raft_consensus.cc:3037] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.751961 14397 raft_consensus.cc:504] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:46.752465 14397 leader_election.cc:288] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:46.752748 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab808742ce8746ae8e9b8fbff435e7be" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:46.752931 12860 raft_consensus.cc:3037] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.754657 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab808742ce8746ae8e9b8fbff435e7be" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:46.754784 12860 raft_consensus.cc:2454] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:46.754827 13120 raft_consensus.cc:3037] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.755148 12664 leader_election.cc:302] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:46.755410 14397 raft_consensus.cc:2781] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:46.755806 14397 raft_consensus.cc:686] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:46.756125 14397 consensus_queue.cc:237] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:46.756907 13120 raft_consensus.cc:2454] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:46.758922 12585 catalog_manager.cc:5484] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:46.917600 14397 raft_consensus.cc:482] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:46.917763 14397 raft_consensus.cc:504] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.918257 14397 leader_election.cc:288] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:46.918715 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3f0742355442af8a29b971921b7d45" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:46.918748 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3f0742355442af8a29b971921b7d45" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:46.918957 12992 raft_consensus.cc:2454] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:46.918957 13120 raft_consensus.cc:2454] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:46.919319 12663 leader_election.cc:302] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:46.919592 14397 raft_consensus.cc:2781] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:46.919682 14397 raft_consensus.cc:482] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:46.919775 14397 raft_consensus.cc:3037] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.921532 14397 raft_consensus.cc:504] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.922052 14397 leader_election.cc:288] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:46.922360 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3f0742355442af8a29b971921b7d45" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:46.922559 13120 raft_consensus.cc:3037] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.922508 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3f0742355442af8a29b971921b7d45" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:46.922690 12992 raft_consensus.cc:3037] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.924774 13120 raft_consensus.cc:2454] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:46.924794 12992 raft_consensus.cc:2454] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:46.925112 12663 leader_election.cc:302] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:46.925346 14397 raft_consensus.cc:2781] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:46.925482 14397 raft_consensus.cc:686] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:46.925736 14397 consensus_queue.cc:237] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.928234 12585 catalog_manager.cc:5484] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:17:46.981496 14450 raft_consensus.cc:482] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:46.981662 14450 raft_consensus.cc:504] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.982192 14450 leader_election.cc:288] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:46.982723 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db051ed0438742319f741a565834b644" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:46.982751 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db051ed0438742319f741a565834b644" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:46.982931 12992 raft_consensus.cc:2454] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:46.982951 12729 raft_consensus.cc:2454] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:46.983319 13056 leader_election.cc:302] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:46.983608 14450 raft_consensus.cc:2781] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:46.983737 14450 raft_consensus.cc:482] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:46.983830 14450 raft_consensus.cc:3037] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.985549 14450 raft_consensus.cc:504] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.986057 14450 leader_election.cc:288] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:46.986490 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db051ed0438742319f741a565834b644" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:46.986548 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db051ed0438742319f741a565834b644" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:46.986667 12729 raft_consensus.cc:3037] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.986692 12992 raft_consensus.cc:3037] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:46.988358 12992 raft_consensus.cc:2454] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:46.988606 12729 raft_consensus.cc:2454] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:46.988734 13056 leader_election.cc:302] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:46.988983 14450 raft_consensus.cc:2781] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:46.989270 14450 raft_consensus.cc:686] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:46.989522 14450 consensus_queue.cc:237] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:46.992959 12585 catalog_manager.cc:5484] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:17:47.279819 14397 consensus_queue.cc:1035] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:47.285065 14397 consensus_queue.cc:1035] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:47.311987 14507 consensus_queue.cc:1035] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:47.316915 14507 consensus_queue.cc:1035] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:47.348480 14512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:47.348760 14512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:47.355820 14512 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 08:17:47.485931 12990 raft_consensus.cc:1261] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:47.486562 14510 consensus_queue.cc:1035] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:47.503070 12729 raft_consensus.cc:1261] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:47.503690 14450 consensus_queue.cc:1035] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 205.159 ms
time per row: 0.10258 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:48.568552 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:48.577739 12708 tablet_service.cc:1467] Processing CreateTablet for tablet e506ca0472444a4686609f2c42d66345 (DEFAULT_TABLE table=rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:48.578069 12840 tablet_service.cc:1467] Processing CreateTablet for tablet e506ca0472444a4686609f2c42d66345 (DEFAULT_TABLE table=rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:48.578235 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e506ca0472444a4686609f2c42d66345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:48.578490 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e506ca0472444a4686609f2c42d66345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:48.579499 12971 tablet_service.cc:1467] Processing CreateTablet for tablet e506ca0472444a4686609f2c42d66345 (DEFAULT_TABLE table=rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:48.579610 12972 tablet_service.cc:1467] Processing CreateTablet for tablet f68020cc7c1346c888c35c0da7b71b94 (DEFAULT_TABLE table=rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:48.579883 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e506ca0472444a4686609f2c42d66345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:48.580210 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68020cc7c1346c888c35c0da7b71b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:48.581331 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7f5bb648c042a5b18d0a04b01ca6a7 (DEFAULT_TABLE table=rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:48.581727 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7f5bb648c042a5b18d0a04b01ca6a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:48.587464 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7f5bb648c042a5b18d0a04b01ca6a7 (DEFAULT_TABLE table=rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:48.587879 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7f5bb648c042a5b18d0a04b01ca6a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:48.588678 12710 tablet_service.cc:1467] Processing CreateTablet for tablet f68020cc7c1346c888c35c0da7b71b94 (DEFAULT_TABLE table=rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:48.589067 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68020cc7c1346c888c35c0da7b71b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:48.591177 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 6a7f5bb648c042a5b18d0a04b01ca6a7 (DEFAULT_TABLE table=rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:48.591559 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a7f5bb648c042a5b18d0a04b01ca6a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:48.594796 12842 tablet_service.cc:1467] Processing CreateTablet for tablet f68020cc7c1346c888c35c0da7b71b94 (DEFAULT_TABLE table=rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:48.594964 14581 tablet_bootstrap.cc:492] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:48.595131 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68020cc7c1346c888c35c0da7b71b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:48.596784 14579 tablet_bootstrap.cc:492] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:48.598379 14579 tablet_bootstrap.cc:654] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:48.604313 14580 tablet_bootstrap.cc:492] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:48.606007 14580 tablet_bootstrap.cc:654] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:48.607249 14582 tablet_bootstrap.cc:492] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:48.608980 14582 tablet_bootstrap.cc:654] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:48.611748 14582 tablet_bootstrap.cc:492] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:48.611948 14582 ts_tablet_manager.cc:1367] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:17:48.612768 14582 raft_consensus.cc:348] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:48.613008 14582 raft_consensus.cc:374] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:48.613152 14582 raft_consensus.cc:729] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:48.613406 14582 consensus_queue.cc:260] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:48.614008 14582 ts_tablet_manager.cc:1398] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:48.614310 14582 tablet_bootstrap.cc:492] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:48.615361 14581 tablet_bootstrap.cc:654] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:48.616529 14582 tablet_bootstrap.cc:654] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:48.620934 14581 tablet_bootstrap.cc:492] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:48.621062 14581 ts_tablet_manager.cc:1367] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.001s
I20240502 08:17:48.621738 14581 raft_consensus.cc:348] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.621933 14581 raft_consensus.cc:374] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:48.622025 14581 raft_consensus.cc:729] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:48.622248 14581 consensus_queue.cc:260] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.622790 14581 ts_tablet_manager.cc:1398] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:48.622960 14582 tablet_bootstrap.cc:492] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:48.623107 14582 ts_tablet_manager.cc:1367] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240502 08:17:48.623778 14582 raft_consensus.cc:348] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.623983 14582 raft_consensus.cc:374] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:48.624065 14582 raft_consensus.cc:729] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:48.624330 14582 consensus_queue.cc:260] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.625108 14580 tablet_bootstrap.cc:492] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:48.625280 14580 ts_tablet_manager.cc:1367] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.004s
I20240502 08:17:48.626359 14580 raft_consensus.cc:348] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.626648 14580 raft_consensus.cc:374] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:48.626752 14580 raft_consensus.cc:729] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:48.626969 14580 consensus_queue.cc:260] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.627278 14579 tablet_bootstrap.cc:492] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:48.627836 14582 ts_tablet_manager.cc:1398] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:17:48.628366 14582 tablet_bootstrap.cc:492] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:48.628422 14579 ts_tablet_manager.cc:1367] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240502 08:17:48.628726 14580 ts_tablet_manager.cc:1398] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:48.628962 14580 tablet_bootstrap.cc:492] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:48.629117 14579 raft_consensus.cc:348] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.629307 14579 raft_consensus.cc:374] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:48.629454 14579 raft_consensus.cc:729] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:48.629653 14579 consensus_queue.cc:260] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.631146 14582 tablet_bootstrap.cc:654] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:48.631311 14579 ts_tablet_manager.cc:1398] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:48.631595 14579 tablet_bootstrap.cc:492] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:48.633154 14579 tablet_bootstrap.cc:654] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:48.633355 14582 tablet_bootstrap.cc:492] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:48.633487 14582 ts_tablet_manager.cc:1367] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:17:48.634093 14580 tablet_bootstrap.cc:654] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:48.634234 14582 raft_consensus.cc:348] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.634423 14582 raft_consensus.cc:374] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:48.634539 14582 raft_consensus.cc:729] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:48.634796 14582 consensus_queue.cc:260] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.635263 14579 tablet_bootstrap.cc:492] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:48.635332 14582 ts_tablet_manager.cc:1398] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:48.635406 14579 ts_tablet_manager.cc:1367] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:48.636065 14579 raft_consensus.cc:348] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:48.636322 14579 raft_consensus.cc:374] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:48.636385 14579 raft_consensus.cc:729] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:48.636610 14579 consensus_queue.cc:260] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:48.637116 14579 ts_tablet_manager.cc:1398] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:48.641724 14580 tablet_bootstrap.cc:492] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:48.642019 14580 ts_tablet_manager.cc:1367] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20240502 08:17:48.642573 14580 raft_consensus.cc:348] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:48.642745 14580 raft_consensus.cc:374] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:48.642815 14580 raft_consensus.cc:729] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:48.643002 14580 consensus_queue.cc:260] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:48.643412 14580 ts_tablet_manager.cc:1398] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:48.643760 14580 tablet_bootstrap.cc:492] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:48.644873 14580 tablet_bootstrap.cc:654] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:48.646387 14580 tablet_bootstrap.cc:492] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:48.646538 14580 ts_tablet_manager.cc:1367] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 08:17:48.647218 14580 raft_consensus.cc:348] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.647394 14580 raft_consensus.cc:374] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:48.647662 14580 raft_consensus.cc:729] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:48.647864 14580 consensus_queue.cc:260] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.648370 14580 ts_tablet_manager.cc:1398] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:48.724639 14573 raft_consensus.cc:482] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:48.724802 14573 raft_consensus.cc:504] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.725555 14573 leader_election.cc:288] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:48.725848 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7f5bb648c042a5b18d0a04b01ca6a7" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:48.726011 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7f5bb648c042a5b18d0a04b01ca6a7" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:48.726100 12861 raft_consensus.cc:2454] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:48.726223 12992 raft_consensus.cc:2454] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:48.726521 13056 leader_election.cc:302] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:48.726779 14573 raft_consensus.cc:2781] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:48.726908 14573 raft_consensus.cc:482] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:48.726996 14573 raft_consensus.cc:3037] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:48.729126 14573 raft_consensus.cc:504] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.729651 14573 leader_election.cc:288] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:48.730020 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7f5bb648c042a5b18d0a04b01ca6a7" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:48.730082 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a7f5bb648c042a5b18d0a04b01ca6a7" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:48.730237 12861 raft_consensus.cc:3037] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:48.730252 12992 raft_consensus.cc:3037] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:48.731175 14497 raft_consensus.cc:482] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:48.731339 14497 raft_consensus.cc:504] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.732142 14497 leader_election.cc:288] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:48.732580 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e506ca0472444a4686609f2c42d66345" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:48.732666 12992 raft_consensus.cc:2454] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:48.732769 12860 raft_consensus.cc:2454] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:48.732823 12861 raft_consensus.cc:2454] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:48.733007 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e506ca0472444a4686609f2c42d66345" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:48.733054 13056 leader_election.cc:302] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:48.733179 12926 leader_election.cc:302] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:48.733208 12729 raft_consensus.cc:2454] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:48.733444 14497 raft_consensus.cc:2781] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:48.733348 14573 raft_consensus.cc:2781] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:48.733557 14497 raft_consensus.cc:482] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:48.733609 14573 raft_consensus.cc:686] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:48.733652 14497 raft_consensus.cc:3037] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:48.734016 14573 consensus_queue.cc:237] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.735685 14497 raft_consensus.cc:504] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.736204 14497 leader_election.cc:288] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:48.736534 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e506ca0472444a4686609f2c42d66345" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:48.736724 12729 raft_consensus.cc:3037] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:48.737157 12579 catalog_manager.cc:5484] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:17:48.738709 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e506ca0472444a4686609f2c42d66345" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:48.738941 12860 raft_consensus.cc:3037] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:48.738934 12729 raft_consensus.cc:2454] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:48.739342 12926 leader_election.cc:302] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:48.739614 14497 raft_consensus.cc:2781] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:48.740414 14497 raft_consensus.cc:686] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:48.740736 14497 consensus_queue.cc:237] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:48.741254 12860 raft_consensus.cc:2454] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:48.743935 12579 catalog_manager.cc:5484] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:17:48.776747 14574 raft_consensus.cc:482] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:48.776944 14574 raft_consensus.cc:504] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:48.777773 14574 leader_election.cc:288] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:48.778322 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68020cc7c1346c888c35c0da7b71b94" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:48.778374 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68020cc7c1346c888c35c0da7b71b94" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:48.778573 12992 raft_consensus.cc:2454] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:48.778676 12860 raft_consensus.cc:2454] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:48.779033 12664 leader_election.cc:302] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:48.779436 14574 raft_consensus.cc:2781] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:48.779561 14574 raft_consensus.cc:482] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:48.779654 14574 raft_consensus.cc:3037] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:48.781468 14574 raft_consensus.cc:504] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:48.782017 14574 leader_election.cc:288] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:48.782398 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68020cc7c1346c888c35c0da7b71b94" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:48.782481 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68020cc7c1346c888c35c0da7b71b94" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:48.782605 12992 raft_consensus.cc:3037] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:48.782661 12860 raft_consensus.cc:3037] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:48.785286 12992 raft_consensus.cc:2454] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:48.785272 12860 raft_consensus.cc:2454] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:48.785647 12664 leader_election.cc:302] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:48.785945 14574 raft_consensus.cc:2781] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:48.786098 14574 raft_consensus.cc:686] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:48.786375 14574 consensus_queue.cc:237] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:48.789426 12579 catalog_manager.cc:5484] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
W20240502 08:17:49.156821 14598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:49.157121 14598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:49.164413 14598 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 08:17:49.274722 14595 consensus_queue.cc:1035] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:49.277933 14574 consensus_queue.cc:1035] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:49.307736 14517 consensus_queue.cc:1035] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:49.311398 14497 consensus_queue.cc:1035] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:49.340713 14573 consensus_queue.cc:1035] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:49.379292 14592 consensus_queue.cc:1035] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 271.609 ms
time per row: 0.135804 ms
I20240502 08:17:49.583932 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:49.593379 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 3c37ff40706043e9adc4da4a6da1947a (DEFAULT_TABLE table=rebalance_test_table_11 [id=02e24b5edb434940bb115fce3e1947fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.593631 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 3c37ff40706043e9adc4da4a6da1947a (DEFAULT_TABLE table=rebalance_test_table_11 [id=02e24b5edb434940bb115fce3e1947fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.593894 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c37ff40706043e9adc4da4a6da1947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.594053 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c37ff40706043e9adc4da4a6da1947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.594352 13101 tablet_service.cc:1467] Processing CreateTablet for tablet a3583c839d364bae819928eca86010ec (DEFAULT_TABLE table=rebalance_test_table_11 [id=02e24b5edb434940bb115fce3e1947fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.594877 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3583c839d364bae819928eca86010ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.598929 14668 tablet_bootstrap.cc:492] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:49.599181 13100 tablet_service.cc:1467] Processing CreateTablet for tablet c6cd36cfde5e480cad86a046d59e8940 (DEFAULT_TABLE table=rebalance_test_table_11 [id=02e24b5edb434940bb115fce3e1947fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.599483 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6cd36cfde5e480cad86a046d59e8940. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.600327 12970 tablet_service.cc:1467] Processing CreateTablet for tablet c6cd36cfde5e480cad86a046d59e8940 (DEFAULT_TABLE table=rebalance_test_table_11 [id=02e24b5edb434940bb115fce3e1947fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.600762 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6cd36cfde5e480cad86a046d59e8940. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.600865 14668 tablet_bootstrap.cc:654] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.602819 12971 tablet_service.cc:1467] Processing CreateTablet for tablet a3583c839d364bae819928eca86010ec (DEFAULT_TABLE table=rebalance_test_table_11 [id=02e24b5edb434940bb115fce3e1947fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.603051 12840 tablet_service.cc:1467] Processing CreateTablet for tablet a3583c839d364bae819928eca86010ec (DEFAULT_TABLE table=rebalance_test_table_11 [id=02e24b5edb434940bb115fce3e1947fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.603235 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3583c839d364bae819928eca86010ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.603375 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3583c839d364bae819928eca86010ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.603346 12710 tablet_service.cc:1467] Processing CreateTablet for tablet 3c37ff40706043e9adc4da4a6da1947a (DEFAULT_TABLE table=rebalance_test_table_11 [id=02e24b5edb434940bb115fce3e1947fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.603717 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c37ff40706043e9adc4da4a6da1947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.607582 12708 tablet_service.cc:1467] Processing CreateTablet for tablet c6cd36cfde5e480cad86a046d59e8940 (DEFAULT_TABLE table=rebalance_test_table_11 [id=02e24b5edb434940bb115fce3e1947fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:49.607950 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6cd36cfde5e480cad86a046d59e8940. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.610394 14668 tablet_bootstrap.cc:492] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:49.610565 14668 ts_tablet_manager.cc:1367] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20240502 08:17:49.611536 14668 raft_consensus.cc:348] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.611858 14668 raft_consensus.cc:374] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.612000 14668 raft_consensus.cc:729] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.612321 14668 consensus_queue.cc:260] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.613137 14668 ts_tablet_manager.cc:1398] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:49.613481 14668 tablet_bootstrap.cc:492] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:49.615388 14668 tablet_bootstrap.cc:654] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.623832 14670 tablet_bootstrap.cc:492] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:49.626595 14670 tablet_bootstrap.cc:654] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.629137 14668 tablet_bootstrap.cc:492] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:49.629384 14668 ts_tablet_manager.cc:1367] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s
I20240502 08:17:49.630261 14668 raft_consensus.cc:348] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.630578 14668 raft_consensus.cc:374] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.630715 14668 raft_consensus.cc:729] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.631050 14668 consensus_queue.cc:260] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.631768 14668 ts_tablet_manager.cc:1398] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:49.632753 14670 tablet_bootstrap.cc:492] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:49.632777 14674 tablet_bootstrap.cc:492] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:49.632808 14673 tablet_bootstrap.cc:492] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:49.632937 14670 ts_tablet_manager.cc:1367] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240502 08:17:49.633694 14670 raft_consensus.cc:348] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.633951 14670 raft_consensus.cc:374] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.634084 14670 raft_consensus.cc:729] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.634368 14670 consensus_queue.cc:260] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.634624 14674 tablet_bootstrap.cc:654] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.635006 14673 tablet_bootstrap.cc:654] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.635921 14670 ts_tablet_manager.cc:1398] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:49.636194 14670 tablet_bootstrap.cc:492] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:49.637647 14670 tablet_bootstrap.cc:654] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.641067 14674 tablet_bootstrap.cc:492] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:49.641250 14674 ts_tablet_manager.cc:1367] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240502 08:17:49.641978 14674 raft_consensus.cc:348] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.642220 14674 raft_consensus.cc:374] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.642344 14674 raft_consensus.cc:729] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.642627 14674 consensus_queue.cc:260] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.645344 14674 ts_tablet_manager.cc:1398] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:17:49.645993 14673 tablet_bootstrap.cc:492] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:49.646932 14674 tablet_bootstrap.cc:492] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:49.647120 14673 ts_tablet_manager.cc:1367] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20240502 08:17:49.649412 14673 raft_consensus.cc:348] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.649618 14673 raft_consensus.cc:374] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.649737 14673 raft_consensus.cc:729] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.649956 14673 consensus_queue.cc:260] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.650503 14673 ts_tablet_manager.cc:1398] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:49.650689 14674 tablet_bootstrap.cc:654] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.650923 14673 tablet_bootstrap.cc:492] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:49.652377 14673 tablet_bootstrap.cc:654] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.655673 14670 tablet_bootstrap.cc:492] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:49.655848 14670 ts_tablet_manager.cc:1367] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20240502 08:17:49.656564 14670 raft_consensus.cc:348] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.656793 14670 raft_consensus.cc:374] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.656874 14670 raft_consensus.cc:729] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.658382 14674 tablet_bootstrap.cc:492] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:49.658560 14674 ts_tablet_manager.cc:1367] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20240502 08:17:49.659262 14674 raft_consensus.cc:348] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.659526 14674 raft_consensus.cc:374] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.659626 14674 raft_consensus.cc:729] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.659929 14674 consensus_queue.cc:260] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.660666 14674 ts_tablet_manager.cc:1398] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:49.661677 14674 tablet_bootstrap.cc:492] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:49.662240 14670 consensus_queue.cc:260] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.663681 14670 ts_tablet_manager.cc:1398] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20240502 08:17:49.665541 14674 tablet_bootstrap.cc:654] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:49.667799 14673 tablet_bootstrap.cc:492] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:49.667963 14673 ts_tablet_manager.cc:1367] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.009s
I20240502 08:17:49.668553 14673 raft_consensus.cc:348] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.668735 14673 raft_consensus.cc:374] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.668812 14673 raft_consensus.cc:729] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.669021 14673 consensus_queue.cc:260] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.669622 14673 ts_tablet_manager.cc:1398] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:49.673580 14674 tablet_bootstrap.cc:492] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:49.673758 14674 ts_tablet_manager.cc:1367] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20240502 08:17:49.674562 14674 raft_consensus.cc:348] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.674744 14674 raft_consensus.cc:374] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:49.674829 14674 raft_consensus.cc:729] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:49.675022 14674 consensus_queue.cc:260] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.675554 14674 ts_tablet_manager.cc:1398] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:49.696462 14595 raft_consensus.cc:482] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:49.696637 14595 raft_consensus.cc:504] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.697450 14595 leader_election.cc:288] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:49.697909 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3583c839d364bae819928eca86010ec" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:49.698112 13121 raft_consensus.cc:2454] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:49.698091 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3583c839d364bae819928eca86010ec" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:49.698326 12860 raft_consensus.cc:2454] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:49.698514 12925 leader_election.cc:302] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:49.698824 14595 raft_consensus.cc:2781] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:49.698921 14595 raft_consensus.cc:482] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:49.699019 14595 raft_consensus.cc:3037] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.700953 14595 raft_consensus.cc:504] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.701583 14595 leader_election.cc:288] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:49.702051 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3583c839d364bae819928eca86010ec" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:49.702051 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3583c839d364bae819928eca86010ec" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:49.702246 13121 raft_consensus.cc:3037] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.702243 12860 raft_consensus.cc:3037] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.704934 13121 raft_consensus.cc:2454] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:49.704934 12860 raft_consensus.cc:2454] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:49.705309 12925 leader_election.cc:302] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:49.705600 14595 raft_consensus.cc:2781] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:49.705776 14595 raft_consensus.cc:686] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:49.706019 14595 consensus_queue.cc:237] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.708942 12579 catalog_manager.cc:5484] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:49.852967 14573 raft_consensus.cc:482] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:49.853124 14573 raft_consensus.cc:504] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.853749 14573 leader_election.cc:288] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:49.854089 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cd36cfde5e480cad86a046d59e8940" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:49.854151 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cd36cfde5e480cad86a046d59e8940" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:49.854311 12730 raft_consensus.cc:2454] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:49.854351 12992 raft_consensus.cc:2454] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:49.854750 13056 leader_election.cc:302] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:49.855000 14573 raft_consensus.cc:2781] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:49.855100 14573 raft_consensus.cc:482] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:49.855182 14573 raft_consensus.cc:3037] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.857092 14573 raft_consensus.cc:504] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.857591 14573 leader_election.cc:288] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:49.858053 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cd36cfde5e480cad86a046d59e8940" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:49.858220 12992 raft_consensus.cc:3037] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.860146 12992 raft_consensus.cc:2454] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:49.860468 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cd36cfde5e480cad86a046d59e8940" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:49.860507 13056 leader_election.cc:302] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:49.860670 12730 raft_consensus.cc:3037] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.860837 14573 raft_consensus.cc:2781] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:49.860991 14573 raft_consensus.cc:686] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:49.861255 14573 consensus_queue.cc:237] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.863214 12730 raft_consensus.cc:2454] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:49.864466 12579 catalog_manager.cc:5484] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:49.934487 14517 raft_consensus.cc:482] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:49.934665 14517 raft_consensus.cc:504] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.935344 14517 leader_election.cc:288] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:49.935676 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c37ff40706043e9adc4da4a6da1947a" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:49.935777 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c37ff40706043e9adc4da4a6da1947a" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:49.935859 12860 raft_consensus.cc:2454] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:49.936041 13121 raft_consensus.cc:2454] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:49.936270 12664 leader_election.cc:302] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:49.936817 14574 raft_consensus.cc:2781] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:49.936935 14574 raft_consensus.cc:482] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:49.937104 14574 raft_consensus.cc:3037] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.939232 14574 raft_consensus.cc:504] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.940018 14574 leader_election.cc:288] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:49.940115 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c37ff40706043e9adc4da4a6da1947a" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:49.940219 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c37ff40706043e9adc4da4a6da1947a" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:49.940304 12861 raft_consensus.cc:3037] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.940398 13121 raft_consensus.cc:3037] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:49.942942 12861 raft_consensus.cc:2454] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:49.943027 13121 raft_consensus.cc:2454] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:49.943337 12664 leader_election.cc:302] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:49.943588 14574 raft_consensus.cc:2781] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:49.943727 14574 raft_consensus.cc:686] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:49.943964 14574 consensus_queue.cc:237] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:49.946806 12579 catalog_manager.cc:5484] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:50.148268 14595 consensus_queue.cc:1035] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:50.154034 14595 consensus_queue.cc:1035] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:50.295464 14685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:50.296139 14685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:17:50.300500 14573 consensus_queue.cc:1035] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:50.305408 14685 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:50.305406 14592 consensus_queue.cc:1035] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_11'
I20240502 08:17:50.347115 14574 consensus_queue.cc:1035] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:50.354595 14574 consensus_queue.cc:1035] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 247.8 ms
time per row: 0.1239 ms
I20240502 08:17:50.701086 12581 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:50.710012 12708 tablet_service.cc:1467] Processing CreateTablet for tablet e10246a0daff4378a48013a25eb17c82 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:50.710191 13101 tablet_service.cc:1467] Processing CreateTablet for tablet e10246a0daff4378a48013a25eb17c82 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:50.710532 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e10246a0daff4378a48013a25eb17c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:50.710646 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e10246a0daff4378a48013a25eb17c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:50.711390 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 3c3b998826974f4e9ba61021af981de9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:50.711817 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c3b998826974f4e9ba61021af981de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:50.714835 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 3c3b998826974f4e9ba61021af981de9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:50.715072 12840 tablet_service.cc:1467] Processing CreateTablet for tablet e10246a0daff4378a48013a25eb17c82 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:50.715312 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c3b998826974f4e9ba61021af981de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:50.715263 12971 tablet_service.cc:1467] Processing CreateTablet for tablet 214793210c6c429b86fd2351c7e52325 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:50.715472 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e10246a0daff4378a48013a25eb17c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:50.715636 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 214793210c6c429b86fd2351c7e52325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:50.715941 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 214793210c6c429b86fd2351c7e52325 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:50.716284 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 214793210c6c429b86fd2351c7e52325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:50.718819 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 3c3b998826974f4e9ba61021af981de9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:50.719142 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c3b998826974f4e9ba61021af981de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:50.719388 12710 tablet_service.cc:1467] Processing CreateTablet for tablet 214793210c6c429b86fd2351c7e52325 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:50.719771 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 214793210c6c429b86fd2351c7e52325. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:50.734710 14752 tablet_bootstrap.cc:492] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:50.734829 14753 tablet_bootstrap.cc:492] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:50.735069 14751 tablet_bootstrap.cc:492] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:50.736657 14753 tablet_bootstrap.cc:654] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:50.736702 14751 tablet_bootstrap.cc:654] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:50.738190 14750 tablet_bootstrap.cc:492] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:50.739115 14752 tablet_bootstrap.cc:654] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:50.739959 14750 tablet_bootstrap.cc:654] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:50.741068 14752 tablet_bootstrap.cc:492] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:50.741220 14752 ts_tablet_manager.cc:1367] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 08:17:50.741694 14750 tablet_bootstrap.cc:492] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:50.741870 14750 ts_tablet_manager.cc:1367] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:50.742080 14752 raft_consensus.cc:348] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.742303 14752 raft_consensus.cc:374] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:50.742439 14752 raft_consensus.cc:729] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:50.742532 14750 raft_consensus.cc:348] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.742746 14750 raft_consensus.cc:374] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:50.742843 14750 raft_consensus.cc:729] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:50.742739 14752 consensus_queue.cc:260] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.743101 14750 consensus_queue.cc:260] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.743294 14752 ts_tablet_manager.cc:1398] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:50.743546 14750 ts_tablet_manager.cc:1398] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:50.743841 14750 tablet_bootstrap.cc:492] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:50.745505 14750 tablet_bootstrap.cc:654] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:50.746834 14752 tablet_bootstrap.cc:492] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:50.748545 14752 tablet_bootstrap.cc:654] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:50.750710 14751 tablet_bootstrap.cc:492] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:50.750917 14751 ts_tablet_manager.cc:1367] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20240502 08:17:50.751936 14751 raft_consensus.cc:348] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.752385 14751 raft_consensus.cc:374] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:50.752535 14751 raft_consensus.cc:729] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:50.753114 14751 consensus_queue.cc:260] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.753760 14751 ts_tablet_manager.cc:1398] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:50.753922 14752 tablet_bootstrap.cc:492] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:50.754084 14751 tablet_bootstrap.cc:492] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:50.754137 14752 ts_tablet_manager.cc:1367] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240502 08:17:50.754951 14752 raft_consensus.cc:348] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.755226 14752 raft_consensus.cc:374] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:50.755336 14752 raft_consensus.cc:729] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:50.755600 14752 consensus_queue.cc:260] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.756167 14751 tablet_bootstrap.cc:654] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:50.756166 14752 ts_tablet_manager.cc:1398] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:50.757449 14753 tablet_bootstrap.cc:492] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:50.757594 14753 ts_tablet_manager.cc:1367] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.005s
I20240502 08:17:50.758325 14753 raft_consensus.cc:348] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.758561 14753 raft_consensus.cc:374] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:50.758643 14753 raft_consensus.cc:729] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:50.758857 14753 consensus_queue.cc:260] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.759389 14753 ts_tablet_manager.cc:1398] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:50.760692 14753 tablet_bootstrap.cc:492] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:50.761112 14751 tablet_bootstrap.cc:492] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:50.761247 14751 ts_tablet_manager.cc:1367] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240502 08:17:50.761929 14751 raft_consensus.cc:348] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.761991 14517 raft_consensus.cc:482] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:50.762185 14751 raft_consensus.cc:374] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:50.762125 14517 raft_consensus.cc:504] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.762336 14751 raft_consensus.cc:729] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:50.762508 14753 tablet_bootstrap.cc:654] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:50.762645 14751 consensus_queue.cc:260] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.763058 14517 leader_election.cc:288] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:50.763335 14751 ts_tablet_manager.cc:1398] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:50.763772 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10246a0daff4378a48013a25eb17c82" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:50.764575 14750 tablet_bootstrap.cc:492] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:50.764583 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10246a0daff4378a48013a25eb17c82" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:50.764838 14750 ts_tablet_manager.cc:1367] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.013s
I20240502 08:17:50.765623 14750 raft_consensus.cc:348] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.765893 14750 raft_consensus.cc:374] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:50.766014 14750 raft_consensus.cc:729] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
W20240502 08:17:50.766167 12663 leader_election.cc:341] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:50.766294 14750 consensus_queue.cc:260] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
W20240502 08:17:50.766803 12664 leader_election.cc:341] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:17:50.766950 12664 leader_election.cc:302] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee; no voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a
I20240502 08:17:50.767028 14750 ts_tablet_manager.cc:1398] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:50.767589 14574 raft_consensus.cc:2726] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:17:50.772708 14753 tablet_bootstrap.cc:492] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:50.772856 14753 ts_tablet_manager.cc:1367] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20240502 08:17:50.773630 14753 raft_consensus.cc:348] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.773875 14753 raft_consensus.cc:374] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:50.773952 14753 raft_consensus.cc:729] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:50.774174 14753 consensus_queue.cc:260] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.774650 14753 ts_tablet_manager.cc:1398] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:50.774865 14753 tablet_bootstrap.cc:492] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:50.776278 14753 tablet_bootstrap.cc:654] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:50.777901 14753 tablet_bootstrap.cc:492] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:50.778033 14753 ts_tablet_manager.cc:1367] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:17:50.778613 14753 raft_consensus.cc:348] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.778743 14753 raft_consensus.cc:374] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:50.778821 14753 raft_consensus.cc:729] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:50.779013 14753 consensus_queue.cc:260] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.779394 14753 ts_tablet_manager.cc:1398] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:17:50.880282 14651 raft_consensus.cc:482] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:50.880440 14651 raft_consensus.cc:504] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.880990 14651 leader_election.cc:288] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:50.881356 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3b998826974f4e9ba61021af981de9" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:50.881479 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3b998826974f4e9ba61021af981de9" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:50.881601 13120 raft_consensus.cc:2454] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:50.881692 12861 raft_consensus.cc:2454] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:50.881982 12925 leader_election.cc:302] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:50.882272 14651 raft_consensus.cc:2781] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:50.882392 14651 raft_consensus.cc:482] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:50.882511 14651 raft_consensus.cc:3037] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:50.884573 14651 raft_consensus.cc:504] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.885085 14651 leader_election.cc:288] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:50.885358 14572 raft_consensus.cc:482] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:50.885391 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3b998826974f4e9ba61021af981de9" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:50.885617 13120 raft_consensus.cc:3037] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:50.885537 14572 raft_consensus.cc:504] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.885653 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3b998826974f4e9ba61021af981de9" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:50.885831 12861 raft_consensus.cc:3037] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:50.886603 14572 leader_election.cc:288] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:50.887553 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "214793210c6c429b86fd2351c7e52325" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:50.887764 12992 raft_consensus.cc:2454] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:50.888222 13120 raft_consensus.cc:2454] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:50.888307 12796 leader_election.cc:302] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:50.888377 12861 raft_consensus.cc:2454] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:50.888531 14572 raft_consensus.cc:2781] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:50.888584 12925 leader_election.cc:302] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:50.888626 14572 raft_consensus.cc:482] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:50.888715 14572 raft_consensus.cc:3037] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:50.888689 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "214793210c6c429b86fd2351c7e52325" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:50.888906 12729 raft_consensus.cc:2454] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:50.889348 14651 raft_consensus.cc:2781] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:50.889576 14651 raft_consensus.cc:686] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:50.889847 14651 consensus_queue.cc:237] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.891448 14572 raft_consensus.cc:504] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.892052 14572 leader_election.cc:288] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:50.893275 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "214793210c6c429b86fd2351c7e52325" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:50.893476 12730 raft_consensus.cc:3037] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:50.893688 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "214793210c6c429b86fd2351c7e52325" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:50.893873 12992 raft_consensus.cc:3037] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:50.895385 12730 raft_consensus.cc:2454] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:50.895675 12992 raft_consensus.cc:2454] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:50.895727 12796 leader_election.cc:302] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:50.896010 14763 raft_consensus.cc:2781] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:50.896162 14763 raft_consensus.cc:686] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:17:50.896454 14763 consensus_queue.cc:237] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:50.896783 12588 catalog_manager.cc:5484] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:50.899726 12581 catalog_manager.cc:5484] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:51.095641 14592 raft_consensus.cc:482] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:51.095790 14592 raft_consensus.cc:504] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:51.096321 14592 leader_election.cc:288] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:51.096629 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10246a0daff4378a48013a25eb17c82" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:51.096774 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10246a0daff4378a48013a25eb17c82" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:51.096845 12730 raft_consensus.cc:2454] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:51.096977 12861 raft_consensus.cc:2454] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:51.097204 13056 leader_election.cc:302] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:51.097476 14592 raft_consensus.cc:2781] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:51.097559 14592 raft_consensus.cc:482] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:51.097658 14592 raft_consensus.cc:3037] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:51.099442 14592 raft_consensus.cc:504] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:51.099925 14592 leader_election.cc:288] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:51.100251 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10246a0daff4378a48013a25eb17c82" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:51.100426 12730 raft_consensus.cc:3037] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:51.100376 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10246a0daff4378a48013a25eb17c82" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:51.100818 12861 raft_consensus.cc:3037] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:51.102416 12730 raft_consensus.cc:2454] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:51.102699 12861 raft_consensus.cc:2454] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:51.102787 13056 leader_election.cc:302] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:51.103183 14592 raft_consensus.cc:2781] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:51.103341 14592 raft_consensus.cc:686] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:51.103597 14592 consensus_queue.cc:237] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:51.106603 12581 catalog_manager.cc:5484] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:51.283012 14595 consensus_queue.cc:1035] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:51.290098 14651 consensus_queue.cc:1035] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:51.501662 14774 consensus_queue.cc:1035] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:51.510394 14774 consensus_queue.cc:1035] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:51.517848 14767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:51.518312 14767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:51.527582 14767 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:51.544272 14592 consensus_queue.cc:1035] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:51.548344 14573 consensus_queue.cc:1035] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_12'
INSERT report
rows total: 2000
time total: 213.928 ms
time per row: 0.106964 ms
I20240502 08:17:51.886225 12588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:51.894310 12971 tablet_service.cc:1467] Processing CreateTablet for tablet 6a72d875fcb54315af393e31e743d2ae (DEFAULT_TABLE table=rebalance_test_table_13 [id=f2c6e6573e254b7d815efdcd481a81a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:51.894804 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a72d875fcb54315af393e31e743d2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:51.894747 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 6a72d875fcb54315af393e31e743d2ae (DEFAULT_TABLE table=rebalance_test_table_13 [id=f2c6e6573e254b7d815efdcd481a81a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:51.895141 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a72d875fcb54315af393e31e743d2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:51.896090 12970 tablet_service.cc:1467] Processing CreateTablet for tablet b87724cd8beb4449b272f972c85a6666 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f2c6e6573e254b7d815efdcd481a81a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:51.896072 12710 tablet_service.cc:1467] Processing CreateTablet for tablet b87724cd8beb4449b272f972c85a6666 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f2c6e6573e254b7d815efdcd481a81a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:51.896447 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b87724cd8beb4449b272f972c85a6666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:51.896453 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b87724cd8beb4449b272f972c85a6666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:51.898901 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 91f185fbf27049a7992daa1d905e5575 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f2c6e6573e254b7d815efdcd481a81a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:51.899264 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f185fbf27049a7992daa1d905e5575. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:51.903406 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 91f185fbf27049a7992daa1d905e5575 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f2c6e6573e254b7d815efdcd481a81a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:51.903800 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f185fbf27049a7992daa1d905e5575. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:51.907377 13101 tablet_service.cc:1467] Processing CreateTablet for tablet b87724cd8beb4449b272f972c85a6666 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f2c6e6573e254b7d815efdcd481a81a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:51.907516 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 6a72d875fcb54315af393e31e743d2ae (DEFAULT_TABLE table=rebalance_test_table_13 [id=f2c6e6573e254b7d815efdcd481a81a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:51.907832 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b87724cd8beb4449b272f972c85a6666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:51.908071 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a72d875fcb54315af393e31e743d2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:51.907399 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 91f185fbf27049a7992daa1d905e5575 (DEFAULT_TABLE table=rebalance_test_table_13 [id=f2c6e6573e254b7d815efdcd481a81a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:51.908531 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f185fbf27049a7992daa1d905e5575. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:51.910876 14833 tablet_bootstrap.cc:492] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:51.912436 14833 tablet_bootstrap.cc:654] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:51.914330 14834 tablet_bootstrap.cc:492] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:51.917438 14834 tablet_bootstrap.cc:654] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:51.931313 14837 tablet_bootstrap.cc:492] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:51.932969 14832 tablet_bootstrap.cc:492] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:51.933130 14833 tablet_bootstrap.cc:492] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:51.933305 14833 ts_tablet_manager.cc:1367] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.000s
I20240502 08:17:51.934000 14837 tablet_bootstrap.cc:654] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:51.934054 14833 raft_consensus.cc:348] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:51.934191 14834 tablet_bootstrap.cc:492] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:51.934319 14834 ts_tablet_manager.cc:1367] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.003s
I20240502 08:17:51.934646 14833 raft_consensus.cc:374] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:51.934798 14833 raft_consensus.cc:729] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:51.934914 14832 tablet_bootstrap.cc:654] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:51.935106 14833 consensus_queue.cc:260] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:51.935678 14833 ts_tablet_manager.cc:1398] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:51.936043 14834 raft_consensus.cc:348] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:51.936672 14833 tablet_bootstrap.cc:492] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:51.937024 14837 tablet_bootstrap.cc:492] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:51.937209 14837 ts_tablet_manager.cc:1367] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 08:17:51.936241 14834 raft_consensus.cc:374] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:51.937500 14834 raft_consensus.cc:729] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:51.937753 14834 consensus_queue.cc:260] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:51.939240 14834 ts_tablet_manager.cc:1398] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:17:51.939198 14837 raft_consensus.cc:348] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:51.939507 14834 tablet_bootstrap.cc:492] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:51.939505 14837 raft_consensus.cc:374] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:51.939666 14837 raft_consensus.cc:729] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:51.939952 14837 consensus_queue.cc:260] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:51.940542 14837 ts_tablet_manager.cc:1398] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:17:51.940910 14833 tablet_bootstrap.cc:654] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:51.941777 14834 tablet_bootstrap.cc:654] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:51.945247 14833 tablet_bootstrap.cc:492] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:51.945385 14833 ts_tablet_manager.cc:1367] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240502 08:17:51.945423 14834 tablet_bootstrap.cc:492] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:51.945560 14834 ts_tablet_manager.cc:1367] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240502 08:17:51.946040 14833 raft_consensus.cc:348] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:51.946200 14833 raft_consensus.cc:374] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:51.946276 14833 raft_consensus.cc:729] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:51.946281 14834 raft_consensus.cc:348] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:51.946518 14834 raft_consensus.cc:374] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:51.946530 14833 consensus_queue.cc:260] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:51.946599 14834 raft_consensus.cc:729] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:51.947157 14833 ts_tablet_manager.cc:1398] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:51.947142 14834 consensus_queue.cc:260] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:51.947365 14832 tablet_bootstrap.cc:492] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:51.947543 14832 ts_tablet_manager.cc:1367] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20240502 08:17:51.948457 14834 ts_tablet_manager.cc:1398] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:51.948726 14834 tablet_bootstrap.cc:492] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:51.949316 14832 raft_consensus.cc:348] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:51.949472 14832 raft_consensus.cc:374] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:51.949589 14832 raft_consensus.cc:729] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:51.949813 14832 consensus_queue.cc:260] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:51.950150 14834 tablet_bootstrap.cc:654] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:51.950496 14832 ts_tablet_manager.cc:1398] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:51.950783 14832 tablet_bootstrap.cc:492] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:51.952324 14832 tablet_bootstrap.cc:654] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:51.953312 14834 tablet_bootstrap.cc:492] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:51.953451 14834 ts_tablet_manager.cc:1367] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 08:17:51.954118 14834 raft_consensus.cc:348] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:51.954278 14834 raft_consensus.cc:374] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:51.954375 14834 raft_consensus.cc:729] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:51.954607 14834 consensus_queue.cc:260] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:51.955188 14834 ts_tablet_manager.cc:1398] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:51.956133 14832 tablet_bootstrap.cc:492] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:51.956282 14832 ts_tablet_manager.cc:1367] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240502 08:17:51.956919 14832 raft_consensus.cc:348] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:51.957109 14832 raft_consensus.cc:374] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:51.957191 14832 raft_consensus.cc:729] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:51.957388 14832 consensus_queue.cc:260] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:51.957825 14832 ts_tablet_manager.cc:1398] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:17:51.958267 14832 tablet_bootstrap.cc:492] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:51.959882 14832 tablet_bootstrap.cc:654] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:51.961277 14832 tablet_bootstrap.cc:492] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:51.961392 14832 ts_tablet_manager.cc:1367] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:51.961962 14832 raft_consensus.cc:348] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:51.962100 14832 raft_consensus.cc:374] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:51.962183 14832 raft_consensus.cc:729] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:51.962373 14832 consensus_queue.cc:260] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:51.962860 14832 ts_tablet_manager.cc:1398] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:17:52.032172 14592 raft_consensus.cc:482] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:52.032325 14592 raft_consensus.cc:504] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:52.032850 14592 leader_election.cc:288] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:52.033214 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f185fbf27049a7992daa1d905e5575" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:52.033282 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f185fbf27049a7992daa1d905e5575" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:52.033430 12861 raft_consensus.cc:2454] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:52.033463 12730 raft_consensus.cc:2454] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:52.033782 13056 leader_election.cc:302] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:52.034040 14592 raft_consensus.cc:2781] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:52.034128 14592 raft_consensus.cc:482] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:52.034224 14592 raft_consensus.cc:3037] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.035979 14592 raft_consensus.cc:504] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:52.036440 14592 leader_election.cc:288] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:52.036770 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f185fbf27049a7992daa1d905e5575" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:52.036841 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f185fbf27049a7992daa1d905e5575" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:52.036942 12861 raft_consensus.cc:3037] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.037036 12730 raft_consensus.cc:3037] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.039126 12861 raft_consensus.cc:2454] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:52.039149 12730 raft_consensus.cc:2454] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:52.039439 13056 leader_election.cc:302] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:52.039500 14773 raft_consensus.cc:482] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:52.039625 14773 raft_consensus.cc:504] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:52.040323 14773 leader_election.cc:288] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:52.040519 14592 raft_consensus.cc:2781] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:52.040688 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a72d875fcb54315af393e31e743d2ae" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:52.040729 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a72d875fcb54315af393e31e743d2ae" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:52.040891 12992 raft_consensus.cc:2454] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:52.040884 13120 raft_consensus.cc:2454] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:52.041157 14592 raft_consensus.cc:686] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:52.041304 12664 leader_election.cc:302] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:52.041461 14592 consensus_queue.cc:237] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:52.041669 14773 raft_consensus.cc:2781] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:52.041771 14773 raft_consensus.cc:482] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:52.041857 14773 raft_consensus.cc:3037] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.044191 14773 raft_consensus.cc:504] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:52.044441 12581 catalog_manager.cc:5484] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:52.045042 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a72d875fcb54315af393e31e743d2ae" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:52.045233 12992 raft_consensus.cc:3037] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.045441 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a72d875fcb54315af393e31e743d2ae" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:52.045612 13120 raft_consensus.cc:3037] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.048015 12992 raft_consensus.cc:2454] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:52.048174 13120 raft_consensus.cc:2454] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:52.048398 12664 leader_election.cc:302] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:52.048655 14715 raft_consensus.cc:2781] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:52.048774 14715 raft_consensus.cc:686] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:52.049118 14715 consensus_queue.cc:237] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:52.050791 14773 leader_election.cc:288] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:52.066226 12581 catalog_manager.cc:5484] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:52.235991 14715 raft_consensus.cc:482] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:52.236143 14715 raft_consensus.cc:504] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:52.236675 14715 leader_election.cc:288] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:52.237018 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b87724cd8beb4449b272f972c85a6666" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:52.237084 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b87724cd8beb4449b272f972c85a6666" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:52.237217 13120 raft_consensus.cc:2454] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:52.237324 12992 raft_consensus.cc:2454] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:52.237669 12664 leader_election.cc:302] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:52.237926 14715 raft_consensus.cc:2781] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:52.238011 14715 raft_consensus.cc:482] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:52.238116 14715 raft_consensus.cc:3037] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.239928 14715 raft_consensus.cc:504] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:52.240481 14715 leader_election.cc:288] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:52.240759 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b87724cd8beb4449b272f972c85a6666" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:52.240854 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b87724cd8beb4449b272f972c85a6666" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:52.240926 13120 raft_consensus.cc:3037] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.241036 12992 raft_consensus.cc:3037] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:52.242692 13120 raft_consensus.cc:2454] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:52.242834 12992 raft_consensus.cc:2454] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:52.243072 12663 leader_election.cc:302] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:52.243352 14715 raft_consensus.cc:2781] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:52.243502 14715 raft_consensus.cc:686] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:52.243741 14715 consensus_queue.cc:237] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:52.246927 12581 catalog_manager.cc:5484] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:17:52.439932 14715 consensus_queue.cc:1035] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:52.454757 14715 consensus_queue.cc:1035] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:52.498926 14592 consensus_queue.cc:1035] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:52.505198 14573 consensus_queue.cc:1035] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:52.606428 14851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:52.606840 14851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:52.614832 14851 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240502 08:17:52.761713 14715 consensus_queue.cc:1035] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:52.812048 12664 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 50077.8us
I20240502 08:17:52.813200 12990 raft_consensus.cc:1261] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:52.816980 14715 consensus_queue.cc:1035] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 275.416 ms
time per row: 0.137708 ms
I20240502 08:17:53.033869 12581 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:53.044235 12971 tablet_service.cc:1467] Processing CreateTablet for tablet 22b69a35c6c6413fb3631c657cfd97f0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.044689 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b69a35c6c6413fb3631c657cfd97f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.046696 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 015aca35bbdf4fb09c8a801d5786522c (DEFAULT_TABLE table=rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.047158 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015aca35bbdf4fb09c8a801d5786522c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.048079 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 22b69a35c6c6413fb3631c657cfd97f0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.048086 12709 tablet_service.cc:1467] Processing CreateTablet for tablet c4438de6e9f14717acba7c9cf279bb4e (DEFAULT_TABLE table=rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.048475 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b69a35c6c6413fb3631c657cfd97f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.048485 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4438de6e9f14717acba7c9cf279bb4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.052165 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 015aca35bbdf4fb09c8a801d5786522c (DEFAULT_TABLE table=rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.052505 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015aca35bbdf4fb09c8a801d5786522c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.054785 12841 tablet_service.cc:1467] Processing CreateTablet for tablet c4438de6e9f14717acba7c9cf279bb4e (DEFAULT_TABLE table=rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.055120 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4438de6e9f14717acba7c9cf279bb4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.058882 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 22b69a35c6c6413fb3631c657cfd97f0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.059258 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b69a35c6c6413fb3631c657cfd97f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.062943 14912 tablet_bootstrap.cc:492] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:53.063688 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 015aca35bbdf4fb09c8a801d5786522c (DEFAULT_TABLE table=rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.064074 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015aca35bbdf4fb09c8a801d5786522c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.064924 14912 tablet_bootstrap.cc:654] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.066839 12972 tablet_service.cc:1467] Processing CreateTablet for tablet c4438de6e9f14717acba7c9cf279bb4e (DEFAULT_TABLE table=rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:53.067234 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4438de6e9f14717acba7c9cf279bb4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:53.087031 14916 tablet_bootstrap.cc:492] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:53.090039 14916 tablet_bootstrap.cc:654] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.091485 14913 tablet_bootstrap.cc:492] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:53.092554 14914 tablet_bootstrap.cc:492] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:53.093022 14912 tablet_bootstrap.cc:492] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:53.093173 14912 ts_tablet_manager.cc:1367] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.006s
I20240502 08:17:53.093992 14912 raft_consensus.cc:348] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:53.094228 14912 raft_consensus.cc:374] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.094311 14912 raft_consensus.cc:729] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.094615 14912 consensus_queue.cc:260] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:53.094928 14914 tablet_bootstrap.cc:654] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.095686 14913 tablet_bootstrap.cc:654] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.096372 14912 ts_tablet_manager.cc:1398] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:53.097892 14916 tablet_bootstrap.cc:492] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:53.098057 14916 ts_tablet_manager.cc:1367] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 08:17:53.098839 14916 raft_consensus.cc:348] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:53.099154 14916 raft_consensus.cc:374] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.099298 14916 raft_consensus.cc:729] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.099625 14916 consensus_queue.cc:260] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:53.103349 14913 tablet_bootstrap.cc:492] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:53.103497 14913 ts_tablet_manager.cc:1367] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240502 08:17:53.103821 14916 ts_tablet_manager.cc:1398] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 08:17:53.104243 14913 raft_consensus.cc:348] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:53.104444 14913 raft_consensus.cc:374] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.104529 14913 raft_consensus.cc:729] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.104763 14916 tablet_bootstrap.cc:492] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:53.104764 14913 consensus_queue.cc:260] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:53.105731 14913 ts_tablet_manager.cc:1398] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:53.106010 14913 tablet_bootstrap.cc:492] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:53.108208 14913 tablet_bootstrap.cc:654] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.108651 14916 tablet_bootstrap.cc:654] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.110795 14914 tablet_bootstrap.cc:492] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:53.110996 14914 ts_tablet_manager.cc:1367] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240502 08:17:53.111843 14914 raft_consensus.cc:348] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:53.112112 14914 raft_consensus.cc:374] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.112236 14914 raft_consensus.cc:729] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.112514 14914 consensus_queue.cc:260] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:53.113138 14914 ts_tablet_manager.cc:1398] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:53.113498 14914 tablet_bootstrap.cc:492] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:53.114436 14916 tablet_bootstrap.cc:492] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:53.114605 14916 ts_tablet_manager.cc:1367] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20240502 08:17:53.115057 14914 tablet_bootstrap.cc:654] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.115279 14916 raft_consensus.cc:348] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:53.115473 14916 raft_consensus.cc:374] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.115540 14916 raft_consensus.cc:729] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.115787 14916 consensus_queue.cc:260] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:53.118810 14916 ts_tablet_manager.cc:1398] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:17:53.119596 14916 tablet_bootstrap.cc:492] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:53.122788 14916 tablet_bootstrap.cc:654] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.124765 14916 tablet_bootstrap.cc:492] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:53.124890 14916 ts_tablet_manager.cc:1367] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:17:53.125243 14913 tablet_bootstrap.cc:492] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:53.125396 14913 ts_tablet_manager.cc:1367] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.015s
I20240502 08:17:53.125530 14916 raft_consensus.cc:348] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:53.125698 14916 raft_consensus.cc:374] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.125784 14916 raft_consensus.cc:729] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.125999 14916 consensus_queue.cc:260] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:53.126102 14913 raft_consensus.cc:348] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:53.126327 14913 raft_consensus.cc:374] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.126387 14913 raft_consensus.cc:729] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.126657 14913 consensus_queue.cc:260] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:53.127162 14913 ts_tablet_manager.cc:1398] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:53.127669 14916 ts_tablet_manager.cc:1398] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:53.129639 14914 tablet_bootstrap.cc:492] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:53.129894 14914 ts_tablet_manager.cc:1367] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240502 08:17:53.130652 14914 raft_consensus.cc:348] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:53.130997 14914 raft_consensus.cc:374] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.131126 14914 raft_consensus.cc:729] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.131434 14914 consensus_queue.cc:260] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:53.131991 14914 ts_tablet_manager.cc:1398] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:53.132306 14914 tablet_bootstrap.cc:492] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:53.133802 14914 tablet_bootstrap.cc:654] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:53.137499 14914 tablet_bootstrap.cc:492] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:53.137643 14914 ts_tablet_manager.cc:1367] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:17:53.138311 14914 raft_consensus.cc:348] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:53.138532 14914 raft_consensus.cc:374] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:53.138630 14914 raft_consensus.cc:729] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:53.138834 14914 consensus_queue.cc:260] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:53.139330 14914 ts_tablet_manager.cc:1398] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:53.202020 14592 raft_consensus.cc:482] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:53.202200 14592 raft_consensus.cc:504] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:53.202950 14592 leader_election.cc:288] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:53.203313 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015aca35bbdf4fb09c8a801d5786522c" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:53.203377 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015aca35bbdf4fb09c8a801d5786522c" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:53.203516 12860 raft_consensus.cc:2454] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:53.203574 12992 raft_consensus.cc:2454] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:53.203938 13056 leader_election.cc:302] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:53.204239 14592 raft_consensus.cc:2781] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:53.204329 14592 raft_consensus.cc:482] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:53.204427 14592 raft_consensus.cc:3037] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.206276 14592 raft_consensus.cc:504] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:53.206935 14592 leader_election.cc:288] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:53.207487 14715 raft_consensus.cc:482] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:53.207620 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015aca35bbdf4fb09c8a801d5786522c" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:53.207633 14715 raft_consensus.cc:504] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:53.207791 12860 raft_consensus.cc:3037] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.208261 14715 leader_election.cc:288] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:53.208902 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b69a35c6c6413fb3631c657cfd97f0" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:53.208971 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b69a35c6c6413fb3631c657cfd97f0" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:53.209102 12861 raft_consensus.cc:2454] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:53.209151 12990 raft_consensus.cc:2454] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:53.209542 12664 leader_election.cc:302] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:53.210005 14715 raft_consensus.cc:2781] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:53.210142 14715 raft_consensus.cc:482] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:53.210227 14715 raft_consensus.cc:3037] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.210364 12860 raft_consensus.cc:2454] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:53.210829 13056 leader_election.cc:302] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:53.211084 14592 raft_consensus.cc:2781] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:53.211237 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015aca35bbdf4fb09c8a801d5786522c" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:53.211431 12992 raft_consensus.cc:3037] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.211279 14592 raft_consensus.cc:686] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:53.211791 14592 consensus_queue.cc:237] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:53.212776 14715 raft_consensus.cc:504] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:53.213362 14715 leader_election.cc:288] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:53.213953 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b69a35c6c6413fb3631c657cfd97f0" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:53.214078 12992 raft_consensus.cc:2454] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:53.214126 12990 raft_consensus.cc:3037] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.214690 14907 raft_consensus.cc:482] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:53.214833 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b69a35c6c6413fb3631c657cfd97f0" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:53.214860 14907 raft_consensus.cc:504] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:53.215050 12860 raft_consensus.cc:3037] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.215617 14907 leader_election.cc:288] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:53.216423 12990 raft_consensus.cc:2454] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:53.216591 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4438de6e9f14717acba7c9cf279bb4e" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:53.216866 12730 raft_consensus.cc:2454] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:53.217371 12664 leader_election.cc:302] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:53.217620 12860 raft_consensus.cc:2454] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:53.217681 14715 raft_consensus.cc:2781] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:53.217746 12796 leader_election.cc:302] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:53.217761 12588 catalog_manager.cc:5484] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:53.218962 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4438de6e9f14717acba7c9cf279bb4e" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:53.219030 14907 raft_consensus.cc:2781] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:53.219164 12990 raft_consensus.cc:2454] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:53.219177 14907 raft_consensus.cc:482] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:53.219282 14907 raft_consensus.cc:3037] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.220577 14715 raft_consensus.cc:686] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:53.220907 14715 consensus_queue.cc:237] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:53.222327 14907 raft_consensus.cc:504] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:53.223049 14907 leader_election.cc:288] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:53.223773 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4438de6e9f14717acba7c9cf279bb4e" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:53.223949 12729 raft_consensus.cc:3037] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.224123 12588 catalog_manager.cc:5484] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:17:53.224853 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4438de6e9f14717acba7c9cf279bb4e" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:53.225033 12990 raft_consensus.cc:3037] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:53.226073 12729 raft_consensus.cc:2454] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:53.226737 12796 leader_election.cc:302] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:53.227277 14907 raft_consensus.cc:2781] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:53.227429 14907 raft_consensus.cc:686] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:17:53.227447 12990 raft_consensus.cc:2454] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:53.227728 14907 consensus_queue.cc:237] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:53.230633 12588 catalog_manager.cc:5484] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
W20240502 08:17:53.593621 14928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:53.594033 14928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:53.602983 14928 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:53.604890 14907 consensus_queue.cc:1035] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.609812 14896 consensus_queue.cc:1035] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.632303 14592 consensus_queue.cc:1035] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.633723 14715 consensus_queue.cc:1035] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.650861 14907 consensus_queue.cc:1035] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:53.661470 14573 consensus_queue.cc:1035] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_14'
INSERT report
rows total: 2000
time total: 200.745 ms
time per row: 0.100373 ms
I20240502 08:17:54.008621 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:54.018720 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8e5440619a4cb0b40dd286862184cd (DEFAULT_TABLE table=rebalance_test_table_15 [id=f13587d289664cc99887ffe5c7242379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.018985 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8e5440619a4cb0b40dd286862184cd (DEFAULT_TABLE table=rebalance_test_table_15 [id=f13587d289664cc99887ffe5c7242379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.019192 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8e5440619a4cb0b40dd286862184cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.019148 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 7a8e5440619a4cb0b40dd286862184cd (DEFAULT_TABLE table=rebalance_test_table_15 [id=f13587d289664cc99887ffe5c7242379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.019397 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8e5440619a4cb0b40dd286862184cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.019603 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8e5440619a4cb0b40dd286862184cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.023412 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 660b389a179f49eeb3007dea6ce078e9 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f13587d289664cc99887ffe5c7242379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.023666 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 660b389a179f49eeb3007dea6ce078e9 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f13587d289664cc99887ffe5c7242379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.023804 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660b389a179f49eeb3007dea6ce078e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.023739 12971 tablet_service.cc:1467] Processing CreateTablet for tablet 660b389a179f49eeb3007dea6ce078e9 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f13587d289664cc99887ffe5c7242379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.024052 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660b389a179f49eeb3007dea6ce078e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.024104 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660b389a179f49eeb3007dea6ce078e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.026902 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 6b40f3044cd6485ca72b05873dc93f7e (DEFAULT_TABLE table=rebalance_test_table_15 [id=f13587d289664cc99887ffe5c7242379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.027220 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b40f3044cd6485ca72b05873dc93f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.030858 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 6b40f3044cd6485ca72b05873dc93f7e (DEFAULT_TABLE table=rebalance_test_table_15 [id=f13587d289664cc99887ffe5c7242379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.031100 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 6b40f3044cd6485ca72b05873dc93f7e (DEFAULT_TABLE table=rebalance_test_table_15 [id=f13587d289664cc99887ffe5c7242379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:54.031219 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b40f3044cd6485ca72b05873dc93f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.031510 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b40f3044cd6485ca72b05873dc93f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:54.038633 14991 tablet_bootstrap.cc:492] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:54.039013 14990 tablet_bootstrap.cc:492] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:54.040529 14991 tablet_bootstrap.cc:654] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.041471 14990 tablet_bootstrap.cc:654] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.049638 14991 tablet_bootstrap.cc:492] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:54.049798 14991 ts_tablet_manager.cc:1367] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240502 08:17:54.050684 14991 raft_consensus.cc:348] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.050907 14991 raft_consensus.cc:374] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.050999 14991 raft_consensus.cc:729] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.051263 14990 tablet_bootstrap.cc:492] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:54.051240 14991 consensus_queue.cc:260] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.051512 14990 ts_tablet_manager.cc:1367] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20240502 08:17:54.051909 14991 ts_tablet_manager.cc:1398] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:54.052259 14990 raft_consensus.cc:348] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.052527 14990 raft_consensus.cc:374] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.052655 14990 raft_consensus.cc:729] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.052739 14991 tablet_bootstrap.cc:492] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:54.052965 14990 consensus_queue.cc:260] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.053625 14990 ts_tablet_manager.cc:1398] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:54.055892 14993 tablet_bootstrap.cc:492] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:54.058334 14990 tablet_bootstrap.cc:492] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:54.059274 14991 tablet_bootstrap.cc:654] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.059957 14990 tablet_bootstrap.cc:654] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.060206 14993 tablet_bootstrap.cc:654] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.063845 14992 tablet_bootstrap.cc:492] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:54.064407 14991 tablet_bootstrap.cc:492] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:54.064559 14991 ts_tablet_manager.cc:1367] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 08:17:54.065279 14991 raft_consensus.cc:348] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.065497 14991 raft_consensus.cc:374] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.065582 14992 tablet_bootstrap.cc:654] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.065584 14991 raft_consensus.cc:729] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.065798 14991 consensus_queue.cc:260] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.066339 14991 ts_tablet_manager.cc:1398] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:54.067343 14993 tablet_bootstrap.cc:492] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:54.067507 14993 ts_tablet_manager.cc:1367] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 08:17:54.068348 14993 raft_consensus.cc:348] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:54.068537 14993 raft_consensus.cc:374] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.068619 14993 raft_consensus.cc:729] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.068848 14993 consensus_queue.cc:260] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:54.069147 14990 tablet_bootstrap.cc:492] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:54.069339 14990 ts_tablet_manager.cc:1367] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 08:17:54.069334 14993 ts_tablet_manager.cc:1398] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:54.071012 14990 raft_consensus.cc:348] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:54.071451 14993 tablet_bootstrap.cc:492] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:54.072041 14990 raft_consensus.cc:374] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.072177 14990 raft_consensus.cc:729] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.072503 14990 consensus_queue.cc:260] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:54.073002 14993 tablet_bootstrap.cc:654] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.075623 14990 ts_tablet_manager.cc:1398] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:17:54.076639 14992 tablet_bootstrap.cc:492] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:54.076875 14992 ts_tablet_manager.cc:1367] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 08:17:54.077809 14992 raft_consensus.cc:348] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:54.078111 14992 raft_consensus.cc:374] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.078258 14992 raft_consensus.cc:729] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.078612 14992 consensus_queue.cc:260] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:54.079169 14992 ts_tablet_manager.cc:1398] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:54.079982 14992 tablet_bootstrap.cc:492] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:54.080005 14993 tablet_bootstrap.cc:492] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:54.080135 14993 ts_tablet_manager.cc:1367] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 08:17:54.080832 14993 raft_consensus.cc:348] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.081066 14993 raft_consensus.cc:374] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.081163 14993 raft_consensus.cc:729] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.081429 14993 consensus_queue.cc:260] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.082087 14993 ts_tablet_manager.cc:1398] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:54.082396 14993 tablet_bootstrap.cc:492] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:54.083552 14992 tablet_bootstrap.cc:654] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.083853 14993 tablet_bootstrap.cc:654] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:54.089490 14992 tablet_bootstrap.cc:492] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:54.089587 14993 tablet_bootstrap.cc:492] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:54.089708 14993 ts_tablet_manager.cc:1367] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240502 08:17:54.089639 14992 ts_tablet_manager.cc:1367] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240502 08:17:54.090389 14993 raft_consensus.cc:348] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.090592 14993 raft_consensus.cc:374] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.090684 14993 raft_consensus.cc:729] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.090739 14992 raft_consensus.cc:348] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.090932 14992 raft_consensus.cc:374] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:54.090888 14993 consensus_queue.cc:260] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.091001 14992 raft_consensus.cc:729] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:54.091198 14992 consensus_queue.cc:260] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.091398 14993 ts_tablet_manager.cc:1398] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:54.091747 14992 ts_tablet_manager.cc:1398] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:54.137140 14715 raft_consensus.cc:482] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:54.137276 14715 raft_consensus.cc:504] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.137804 14715 leader_election.cc:288] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:54.138154 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b40f3044cd6485ca72b05873dc93f7e" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:54.138234 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b40f3044cd6485ca72b05873dc93f7e" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:54.138363 13121 raft_consensus.cc:2454] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:54.138412 12990 raft_consensus.cc:2454] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:54.138845 12664 leader_election.cc:302] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:54.139110 14715 raft_consensus.cc:2781] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:54.139214 14715 raft_consensus.cc:482] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:54.139268 14715 raft_consensus.cc:3037] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.141012 14715 raft_consensus.cc:504] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.141508 14715 leader_election.cc:288] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:54.141861 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b40f3044cd6485ca72b05873dc93f7e" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:54.141968 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b40f3044cd6485ca72b05873dc93f7e" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:54.142045 13121 raft_consensus.cc:3037] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.142130 12990 raft_consensus.cc:3037] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.144750 13121 raft_consensus.cc:2454] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:54.144750 12990 raft_consensus.cc:2454] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:54.145174 12663 leader_election.cc:302] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:54.145468 14715 raft_consensus.cc:2781] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:54.145603 14715 raft_consensus.cc:686] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:54.145872 14715 consensus_queue.cc:237] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.148876 12579 catalog_manager.cc:5484] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:17:54.179466 14952 raft_consensus.cc:482] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:54.179639 14952 raft_consensus.cc:504] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.180297 14952 leader_election.cc:288] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:54.180655 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a8e5440619a4cb0b40dd286862184cd" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:54.180855 12729 raft_consensus.cc:2454] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:54.180804 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a8e5440619a4cb0b40dd286862184cd" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:54.181000 12861 raft_consensus.cc:2454] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:17:54.181344 13056 leader_election.cc:302] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:54.181605 14952 raft_consensus.cc:2781] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:54.181715 14952 raft_consensus.cc:482] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:54.181813 14952 raft_consensus.cc:3037] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.183940 14952 raft_consensus.cc:504] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.184398 14952 leader_election.cc:288] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:54.184840 12729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a8e5440619a4cb0b40dd286862184cd" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:54.184852 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a8e5440619a4cb0b40dd286862184cd" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:54.185007 12729 raft_consensus.cc:3037] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.185021 12861 raft_consensus.cc:3037] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.187021 12729 raft_consensus.cc:2454] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:54.187719 12861 raft_consensus.cc:2454] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:17:54.188385 13056 leader_election.cc:302] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:54.188663 14952 raft_consensus.cc:2781] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:54.188860 14595 raft_consensus.cc:482] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:54.189016 14595 raft_consensus.cc:504] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:54.189569 14595 leader_election.cc:288] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:54.190107 14952 raft_consensus.cc:686] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:17:54.190415 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660b389a179f49eeb3007dea6ce078e9" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:54.190387 14952 consensus_queue.cc:237] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:54.190642 12861 raft_consensus.cc:2454] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:54.190912 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660b389a179f49eeb3007dea6ce078e9" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:54.191109 13121 raft_consensus.cc:2454] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:54.191452 12925 leader_election.cc:302] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:54.191713 14595 raft_consensus.cc:2781] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:54.191812 14595 raft_consensus.cc:482] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:54.191900 14595 raft_consensus.cc:3037] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.194207 14595 raft_consensus.cc:504] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:54.195565 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660b389a179f49eeb3007dea6ce078e9" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:54.195742 13121 raft_consensus.cc:3037] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.196419 14595 leader_election.cc:288] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:54.197979 12579 catalog_manager.cc:5484] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:17:54.198688 13121 raft_consensus.cc:2454] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:54.199012 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660b389a179f49eeb3007dea6ce078e9" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:54.199119 12925 leader_election.cc:302] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:54.199201 12861 raft_consensus.cc:3037] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:54.199385 14595 raft_consensus.cc:2781] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:54.199609 14595 raft_consensus.cc:686] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:54.199843 14595 consensus_queue.cc:237] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:54.201877 12861 raft_consensus.cc:2454] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:54.203591 12579 catalog_manager.cc:5484] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
W20240502 08:17:54.541977 15006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:54.542248 15006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:54.550000 15006 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240502 08:17:54.631631 14715 consensus_queue.cc:1035] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.635249 14592 consensus_queue.cc:1035] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.678978 14715 consensus_queue.cc:1035] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.696245 14952 consensus_queue.cc:1035] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.720408 14595 consensus_queue.cc:1035] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:54.760636 14857 consensus_queue.cc:1035] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 361.528 ms
time per row: 0.180764 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:55.919456 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:55.928862 12840 tablet_service.cc:1467] Processing CreateTablet for tablet fdc2defcad4c45efa0937e108ce6ad6c (DEFAULT_TABLE table=rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.929033 12709 tablet_service.cc:1467] Processing CreateTablet for tablet fdc2defcad4c45efa0937e108ce6ad6c (DEFAULT_TABLE table=rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.929224 13101 tablet_service.cc:1467] Processing CreateTablet for tablet fdc2defcad4c45efa0937e108ce6ad6c (DEFAULT_TABLE table=rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.929342 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc2defcad4c45efa0937e108ce6ad6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.929383 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc2defcad4c45efa0937e108ce6ad6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.929641 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc2defcad4c45efa0937e108ce6ad6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.933092 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 4047e22c67b8475ca3185e491e37ea98 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.933190 12971 tablet_service.cc:1467] Processing CreateTablet for tablet 25e96ee3f53248b6b461938fb240fc4e (DEFAULT_TABLE table=rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.933440 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4047e22c67b8475ca3185e491e37ea98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.933589 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e96ee3f53248b6b461938fb240fc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.936925 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 25e96ee3f53248b6b461938fb240fc4e (DEFAULT_TABLE table=rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.937325 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e96ee3f53248b6b461938fb240fc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.942817 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 4047e22c67b8475ca3185e491e37ea98 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.943161 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4047e22c67b8475ca3185e491e37ea98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.946149 15074 tablet_bootstrap.cc:492] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:55.946520 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 4047e22c67b8475ca3185e491e37ea98 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.946859 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4047e22c67b8475ca3185e491e37ea98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.947227 15077 tablet_bootstrap.cc:492] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:55.949294 15074 tablet_bootstrap.cc:654] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.949892 15077 tablet_bootstrap.cc:654] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.952199 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 25e96ee3f53248b6b461938fb240fc4e (DEFAULT_TABLE table=rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:55.952617 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e96ee3f53248b6b461938fb240fc4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:55.957614 15074 tablet_bootstrap.cc:492] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:55.957960 15074 ts_tablet_manager.cc:1367] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240502 08:17:55.958778 15074 raft_consensus.cc:348] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:55.959086 15074 raft_consensus.cc:374] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.959494 15074 raft_consensus.cc:729] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.959825 15074 consensus_queue.cc:260] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:55.960451 15074 ts_tablet_manager.cc:1398] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:55.972386 15081 tablet_bootstrap.cc:492] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:55.975831 15074 tablet_bootstrap.cc:492] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:55.977552 15074 tablet_bootstrap.cc:654] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.980532 15075 tablet_bootstrap.cc:492] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:55.981477 15077 tablet_bootstrap.cc:492] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:55.981673 15077 ts_tablet_manager.cc:1367] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.021s
I20240502 08:17:55.982164 15075 tablet_bootstrap.cc:654] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.982425 15081 tablet_bootstrap.cc:654] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.982445 15077 raft_consensus.cc:348] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:55.982798 15077 raft_consensus.cc:374] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.982936 15077 raft_consensus.cc:729] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.983199 15077 consensus_queue.cc:260] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:55.983829 15077 ts_tablet_manager.cc:1398] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:55.984238 15077 tablet_bootstrap.cc:492] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:55.984652 15074 tablet_bootstrap.cc:492] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:55.984841 15074 ts_tablet_manager.cc:1367] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240502 08:17:55.985635 15074 raft_consensus.cc:348] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:55.985862 15077 tablet_bootstrap.cc:654] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.985868 15074 raft_consensus.cc:374] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.985934 15074 raft_consensus.cc:729] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.986160 15074 consensus_queue.cc:260] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:55.986740 15074 ts_tablet_manager.cc:1398] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:17:55.991880 15081 tablet_bootstrap.cc:492] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:55.992084 15081 ts_tablet_manager.cc:1367] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240502 08:17:55.993157 15081 raft_consensus.cc:348] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:55.993351 15081 raft_consensus.cc:374] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.993454 15081 raft_consensus.cc:729] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.994040 15075 tablet_bootstrap.cc:492] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:55.994216 15075 ts_tablet_manager.cc:1367] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240502 08:17:55.994053 15081 consensus_queue.cc:260] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:55.994985 15075 raft_consensus.cc:348] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:55.995201 15075 raft_consensus.cc:374] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.995304 15075 raft_consensus.cc:729] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:55.995541 15081 ts_tablet_manager.cc:1398] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:55.995510 15075 consensus_queue.cc:260] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:55.995980 15081 tablet_bootstrap.cc:492] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:55.996042 15075 ts_tablet_manager.cc:1398] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:55.996621 15075 tablet_bootstrap.cc:492] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:55.998291 15075 tablet_bootstrap.cc:654] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:55.998524 15077 tablet_bootstrap.cc:492] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:55.998817 15077 ts_tablet_manager.cc:1367] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.001s
I20240502 08:17:55.999569 15077 raft_consensus.cc:348] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:55.999830 15077 raft_consensus.cc:374] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:55.999960 15077 raft_consensus.cc:729] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.000234 15077 consensus_queue.cc:260] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:56.000926 15077 ts_tablet_manager.cc:1398] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:56.003577 15081 tablet_bootstrap.cc:654] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.005620 15081 tablet_bootstrap.cc:492] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:56.006004 15081 ts_tablet_manager.cc:1367] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 08:17:56.008216 15081 raft_consensus.cc:348] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:56.008514 15081 raft_consensus.cc:374] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.008625 15081 raft_consensus.cc:729] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.009002 15081 consensus_queue.cc:260] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:56.009646 15081 ts_tablet_manager.cc:1398] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:56.009780 15075 tablet_bootstrap.cc:492] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:56.009945 15075 ts_tablet_manager.cc:1367] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.005s
I20240502 08:17:56.010738 15075 raft_consensus.cc:348] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:56.010929 15075 raft_consensus.cc:374] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.011005 15075 raft_consensus.cc:729] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.011263 15075 consensus_queue.cc:260] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:56.012012 15075 ts_tablet_manager.cc:1398] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:56.012310 15075 tablet_bootstrap.cc:492] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:56.013777 15075 tablet_bootstrap.cc:654] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:56.015883 15075 tablet_bootstrap.cc:492] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:56.016027 15075 ts_tablet_manager.cc:1367] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:17:56.016780 15075 raft_consensus.cc:348] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:56.016978 15075 raft_consensus.cc:374] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:56.017078 15075 raft_consensus.cc:729] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:56.017289 15075 consensus_queue.cc:260] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:56.017796 15075 ts_tablet_manager.cc:1398] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:56.057026 14595 raft_consensus.cc:482] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:56.057180 14595 raft_consensus.cc:504] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:56.057693 14595 leader_election.cc:288] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:56.058005 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e96ee3f53248b6b461938fb240fc4e" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:56.058233 12860 raft_consensus.cc:2454] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:56.058157 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e96ee3f53248b6b461938fb240fc4e" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:56.058406 12730 raft_consensus.cc:2454] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:56.058621 12926 leader_election.cc:302] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:56.058893 14595 raft_consensus.cc:2781] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:56.059022 14595 raft_consensus.cc:482] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:56.059152 14595 raft_consensus.cc:3037] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.061129 14595 raft_consensus.cc:504] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:56.061626 14595 leader_election.cc:288] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:56.062091 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e96ee3f53248b6b461938fb240fc4e" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:56.062160 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e96ee3f53248b6b461938fb240fc4e" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:56.062307 12860 raft_consensus.cc:3037] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.062292 12730 raft_consensus.cc:3037] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.064265 12860 raft_consensus.cc:2454] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:56.064273 12730 raft_consensus.cc:2454] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:56.064620 12926 leader_election.cc:302] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:56.064865 14595 raft_consensus.cc:2781] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:56.065016 14595 raft_consensus.cc:686] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:56.065239 14595 consensus_queue.cc:237] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:56.068084 12588 catalog_manager.cc:5484] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:17:56.195552 15068 raft_consensus.cc:482] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:56.195729 15068 raft_consensus.cc:504] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:56.196398 15068 leader_election.cc:288] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:56.196832 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc2defcad4c45efa0937e108ce6ad6c" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:56.196832 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc2defcad4c45efa0937e108ce6ad6c" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:56.197037 12730 raft_consensus.cc:2454] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:56.197037 13120 raft_consensus.cc:2454] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:56.197400 12795 leader_election.cc:302] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:56.197643 15068 raft_consensus.cc:2781] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:56.197748 15068 raft_consensus.cc:482] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:56.197840 15068 raft_consensus.cc:3037] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.199995 15068 raft_consensus.cc:504] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:56.200490 15068 leader_election.cc:288] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:56.200884 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc2defcad4c45efa0937e108ce6ad6c" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:56.200928 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc2defcad4c45efa0937e108ce6ad6c" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:56.201086 13120 raft_consensus.cc:3037] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.201084 12730 raft_consensus.cc:3037] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.203812 13120 raft_consensus.cc:2454] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:56.203812 12730 raft_consensus.cc:2454] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:56.204209 12795 leader_election.cc:302] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:56.204531 15068 raft_consensus.cc:2781] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:56.204665 15068 raft_consensus.cc:686] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:17:56.204947 15068 consensus_queue.cc:237] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:56.209622 12588 catalog_manager.cc:5484] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:56.277033 14595 raft_consensus.cc:482] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:56.277230 14595 raft_consensus.cc:504] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:56.278014 14595 leader_election.cc:288] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:56.278268 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4047e22c67b8475ca3185e491e37ea98" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:56.278292 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4047e22c67b8475ca3185e491e37ea98" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:56.278600 12860 raft_consensus.cc:2454] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:56.278817 13120 raft_consensus.cc:2454] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:56.279127 12926 leader_election.cc:302] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:56.279615 14595 raft_consensus.cc:2781] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:56.279731 14595 raft_consensus.cc:482] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:56.279815 14595 raft_consensus.cc:3037] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.281716 14595 raft_consensus.cc:504] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:56.282260 14595 leader_election.cc:288] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:56.282562 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4047e22c67b8475ca3185e491e37ea98" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:56.282609 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4047e22c67b8475ca3185e491e37ea98" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:56.282733 13120 raft_consensus.cc:3037] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.282824 12860 raft_consensus.cc:3037] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:56.284564 13120 raft_consensus.cc:2454] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:56.284674 12860 raft_consensus.cc:2454] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:56.284914 12925 leader_election.cc:302] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:56.285259 14595 raft_consensus.cc:2781] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:56.285384 14595 raft_consensus.cc:686] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:56.285840 14595 consensus_queue.cc:237] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:56.288863 12588 catalog_manager.cc:5484] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:56.494035 15076 consensus_queue.cc:1035] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:56.510283 15076 consensus_queue.cc:1035] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:56.654106 15094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:56.654426 15094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:56.662546 15094 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240502 08:17:56.743077 15076 consensus_queue.cc:1035] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:56.750548 14595 consensus_queue.cc:1035] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:56.776654 14779 consensus_queue.cc:1035] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:56.807204 15068 consensus_queue.cc:1035] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 225.996 ms
time per row: 0.112998 ms
I20240502 08:17:57.033991 12580 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:57.045373 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 553cec4eb28341dba84afc79b540ae4b (DEFAULT_TABLE table=rebalance_test_table_17 [id=c8304b52e21948eba4b4290948c74b5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.045814 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553cec4eb28341dba84afc79b540ae4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.046046 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 553cec4eb28341dba84afc79b540ae4b (DEFAULT_TABLE table=rebalance_test_table_17 [id=c8304b52e21948eba4b4290948c74b5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.046437 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553cec4eb28341dba84afc79b540ae4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.046453 12708 tablet_service.cc:1467] Processing CreateTablet for tablet e249007ce4284fd2bc026ab9d6c5d4cb (DEFAULT_TABLE table=rebalance_test_table_17 [id=c8304b52e21948eba4b4290948c74b5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.046447 13101 tablet_service.cc:1467] Processing CreateTablet for tablet e249007ce4284fd2bc026ab9d6c5d4cb (DEFAULT_TABLE table=rebalance_test_table_17 [id=c8304b52e21948eba4b4290948c74b5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.046845 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e249007ce4284fd2bc026ab9d6c5d4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.046870 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e249007ce4284fd2bc026ab9d6c5d4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.050794 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 553cec4eb28341dba84afc79b540ae4b (DEFAULT_TABLE table=rebalance_test_table_17 [id=c8304b52e21948eba4b4290948c74b5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.051200 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553cec4eb28341dba84afc79b540ae4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.056385 13100 tablet_service.cc:1467] Processing CreateTablet for tablet 1fba813ac35c4382a0917604c2057496 (DEFAULT_TABLE table=rebalance_test_table_17 [id=c8304b52e21948eba4b4290948c74b5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.056731 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fba813ac35c4382a0917604c2057496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.057411 15159 tablet_bootstrap.cc:492] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:57.058193 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 1fba813ac35c4382a0917604c2057496 (DEFAULT_TABLE table=rebalance_test_table_17 [id=c8304b52e21948eba4b4290948c74b5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.058990 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fba813ac35c4382a0917604c2057496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.058964 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 1fba813ac35c4382a0917604c2057496 (DEFAULT_TABLE table=rebalance_test_table_17 [id=c8304b52e21948eba4b4290948c74b5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.059229 15159 tablet_bootstrap.cc:654] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.059355 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fba813ac35c4382a0917604c2057496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.061667 15158 tablet_bootstrap.cc:492] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:57.064291 15158 tablet_bootstrap.cc:654] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.064308 12971 tablet_service.cc:1467] Processing CreateTablet for tablet e249007ce4284fd2bc026ab9d6c5d4cb (DEFAULT_TABLE table=rebalance_test_table_17 [id=c8304b52e21948eba4b4290948c74b5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:57.064713 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e249007ce4284fd2bc026ab9d6c5d4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:57.066570 15161 tablet_bootstrap.cc:492] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:57.066991 15159 tablet_bootstrap.cc:492] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:57.067217 15159 ts_tablet_manager.cc:1367] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240502 08:17:57.068349 15159 raft_consensus.cc:348] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:57.068477 15161 tablet_bootstrap.cc:654] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.068640 15159 raft_consensus.cc:374] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.068769 15159 raft_consensus.cc:729] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.069056 15159 consensus_queue.cc:260] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:57.069693 15159 ts_tablet_manager.cc:1398] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:57.069986 15159 tablet_bootstrap.cc:492] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:57.073607 15164 tablet_bootstrap.cc:492] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:57.073779 15158 tablet_bootstrap.cc:492] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:57.073926 15159 tablet_bootstrap.cc:654] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.075551 15158 ts_tablet_manager.cc:1367] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 08:17:57.076309 15164 tablet_bootstrap.cc:654] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.076468 15158 raft_consensus.cc:348] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.079250 15158 raft_consensus.cc:374] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.079435 15158 raft_consensus.cc:729] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.079658 15158 consensus_queue.cc:260] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.080310 15158 ts_tablet_manager.cc:1398] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:17:57.080626 15158 tablet_bootstrap.cc:492] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:57.081400 15161 tablet_bootstrap.cc:492] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:57.081903 15161 ts_tablet_manager.cc:1367] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20240502 08:17:57.082747 15161 raft_consensus.cc:348] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.083340 15161 raft_consensus.cc:374] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.083490 15158 tablet_bootstrap.cc:654] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.083482 15161 raft_consensus.cc:729] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.083788 15161 consensus_queue.cc:260] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.085948 15161 ts_tablet_manager.cc:1398] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:17:57.086246 15161 tablet_bootstrap.cc:492] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:57.086750 15159 tablet_bootstrap.cc:492] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:57.087006 15159 ts_tablet_manager.cc:1367] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.003s
I20240502 08:17:57.087563 15158 tablet_bootstrap.cc:492] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:57.087760 15158 ts_tablet_manager.cc:1367] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 08:17:57.087774 15159 raft_consensus.cc:348] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.088037 15159 raft_consensus.cc:374] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.088161 15159 raft_consensus.cc:729] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.088451 15159 consensus_queue.cc:260] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.089054 15159 ts_tablet_manager.cc:1398] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:57.090286 15158 raft_consensus.cc:348] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:57.090482 15158 raft_consensus.cc:374] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.090562 15158 raft_consensus.cc:729] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.090772 15158 consensus_queue.cc:260] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:57.091305 15161 tablet_bootstrap.cc:654] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.091372 15158 ts_tablet_manager.cc:1398] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:17:57.091761 15158 tablet_bootstrap.cc:492] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:57.092736 15159 tablet_bootstrap.cc:492] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:57.094115 15159 tablet_bootstrap.cc:654] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.094123 15158 tablet_bootstrap.cc:654] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:57.100559 15159 tablet_bootstrap.cc:492] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:57.100620 15164 tablet_bootstrap.cc:492] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:57.100697 15159 ts_tablet_manager.cc:1367] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240502 08:17:57.100935 15164 ts_tablet_manager.cc:1367] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.004s
I20240502 08:17:57.101358 15159 raft_consensus.cc:348] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.101542 15159 raft_consensus.cc:374] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.101625 15159 raft_consensus.cc:729] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.101830 15159 consensus_queue.cc:260] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.102345 15159 ts_tablet_manager.cc:1398] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:57.101970 15164 raft_consensus.cc:348] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.102434 15164 raft_consensus.cc:374] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.102557 15164 raft_consensus.cc:729] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.103045 15164 consensus_queue.cc:260] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.110095 15158 tablet_bootstrap.cc:492] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:57.110237 15158 ts_tablet_manager.cc:1367] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.013s
I20240502 08:17:57.103753 15164 ts_tablet_manager.cc:1398] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:17:57.110989 15158 raft_consensus.cc:348] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.111635 15158 raft_consensus.cc:374] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.111737 15158 raft_consensus.cc:729] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.111968 15158 consensus_queue.cc:260] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.112505 15158 ts_tablet_manager.cc:1398] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:57.118775 15161 tablet_bootstrap.cc:492] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:57.118938 15161 ts_tablet_manager.cc:1367] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.015s
I20240502 08:17:57.119663 15161 raft_consensus.cc:348] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:57.119876 15161 raft_consensus.cc:374] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:57.119954 15161 raft_consensus.cc:729] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:57.120154 15161 consensus_queue.cc:260] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:57.120573 15161 ts_tablet_manager.cc:1398] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:57.225513 15069 raft_consensus.cc:482] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:57.225666 15069 raft_consensus.cc:504] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.226401 15069 leader_election.cc:288] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:57.226717 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fba813ac35c4382a0917604c2057496" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:57.226922 12860 raft_consensus.cc:2454] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:57.226936 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fba813ac35c4382a0917604c2057496" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:57.227138 13121 raft_consensus.cc:2454] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:57.227331 12664 leader_election.cc:302] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:57.227663 15069 raft_consensus.cc:2781] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:57.227775 15069 raft_consensus.cc:482] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:57.227862 15069 raft_consensus.cc:3037] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.229771 15069 raft_consensus.cc:504] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.230258 15069 leader_election.cc:288] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:57.230608 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fba813ac35c4382a0917604c2057496" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:57.230767 12860 raft_consensus.cc:3037] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.230727 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fba813ac35c4382a0917604c2057496" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:57.230887 13121 raft_consensus.cc:3037] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.232563 13121 raft_consensus.cc:2454] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:57.232640 12860 raft_consensus.cc:2454] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:57.232898 12663 leader_election.cc:302] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:57.233165 15069 raft_consensus.cc:2781] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:57.233300 15069 raft_consensus.cc:686] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:57.233559 15069 consensus_queue.cc:237] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.236471 15069 raft_consensus.cc:482] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:57.236639 15069 raft_consensus.cc:504] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:57.236877 12580 catalog_manager.cc:5484] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:57.237180 15069 leader_election.cc:288] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:57.237483 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e249007ce4284fd2bc026ab9d6c5d4cb" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:57.237692 13121 raft_consensus.cc:2454] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:57.237684 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e249007ce4284fd2bc026ab9d6c5d4cb" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:57.237905 12992 raft_consensus.cc:2454] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:57.238032 12663 leader_election.cc:302] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:57.238317 15069 raft_consensus.cc:2781] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:57.238412 15069 raft_consensus.cc:482] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:57.238526 15069 raft_consensus.cc:3037] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.240451 15069 raft_consensus.cc:504] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:57.240974 15069 leader_election.cc:288] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:17:57.241269 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e249007ce4284fd2bc026ab9d6c5d4cb" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:57.241371 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e249007ce4284fd2bc026ab9d6c5d4cb" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:57.241465 13121 raft_consensus.cc:3037] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.241567 12992 raft_consensus.cc:3037] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.244145 13121 raft_consensus.cc:2454] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:57.244153 12992 raft_consensus.cc:2454] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:57.244513 12663 leader_election.cc:302] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:57.244755 15069 raft_consensus.cc:2781] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:57.244916 15069 raft_consensus.cc:686] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:57.245155 15069 consensus_queue.cc:237] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:17:57.247888 12580 catalog_manager.cc:5484] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:17:57.304039 14595 raft_consensus.cc:482] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:57.304189 14595 raft_consensus.cc:504] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.304814 14595 leader_election.cc:288] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:57.305121 12728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553cec4eb28341dba84afc79b540ae4b" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:57.305222 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553cec4eb28341dba84afc79b540ae4b" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:57.305295 12728 raft_consensus.cc:2454] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:57.305425 13121 raft_consensus.cc:2454] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:57.305672 12926 leader_election.cc:302] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:57.305960 14595 raft_consensus.cc:2781] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:57.306080 14595 raft_consensus.cc:482] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:57.306150 14595 raft_consensus.cc:3037] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.307938 14595 raft_consensus.cc:504] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.308426 14595 leader_election.cc:288] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:57.308818 12728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553cec4eb28341dba84afc79b540ae4b" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:57.308840 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553cec4eb28341dba84afc79b540ae4b" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:57.308974 12728 raft_consensus.cc:3037] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.309003 13121 raft_consensus.cc:3037] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:57.311419 12728 raft_consensus.cc:2454] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:57.311559 13121 raft_consensus.cc:2454] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:57.311956 12925 leader_election.cc:302] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:57.312186 14595 raft_consensus.cc:2781] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:57.312357 14595 raft_consensus.cc:686] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:57.312613 14595 consensus_queue.cc:237] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:57.315681 12580 catalog_manager.cc:5484] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:57.655036 14715 consensus_queue.cc:1035] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:57.664958 15069 consensus_queue.cc:1035] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:57.689455 15175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:57.689759 15175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:57.697229 15175 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:57.698220 15076 consensus_queue.cc:1035] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:57.703208 15076 consensus_queue.cc:1035] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_17'
I20240502 08:17:57.749758 14715 consensus_queue.cc:1035] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:57.755162 14715 consensus_queue.cc:1035] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 237.486 ms
time per row: 0.118743 ms
-----------------------------------------------------
Suppressions used:
count bytes template
16 4220 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:17:58.984678 12582 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:17:58.993666 12971 tablet_service.cc:1467] Processing CreateTablet for tablet 1319f55da0b348ac8a325e7d04f0a11d (DEFAULT_TABLE table=rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.994112 12971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1319f55da0b348ac8a325e7d04f0a11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.994812 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 385630fa0b3d4fa7b48af2aa7852d2d5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.995229 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385630fa0b3d4fa7b48af2aa7852d2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.995149 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 1319f55da0b348ac8a325e7d04f0a11d (DEFAULT_TABLE table=rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.995318 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 1319f55da0b348ac8a325e7d04f0a11d (DEFAULT_TABLE table=rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.995576 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1319f55da0b348ac8a325e7d04f0a11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.995703 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1319f55da0b348ac8a325e7d04f0a11d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.998804 12842 tablet_service.cc:1467] Processing CreateTablet for tablet ed9fd50fd4474d25883861f03d6333c0 (DEFAULT_TABLE table=rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.999177 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9fd50fd4474d25883861f03d6333c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.994112 12970 tablet_service.cc:1467] Processing CreateTablet for tablet ed9fd50fd4474d25883861f03d6333c0 (DEFAULT_TABLE table=rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:58.999614 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9fd50fd4474d25883861f03d6333c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.001087 13102 tablet_service.cc:1467] Processing CreateTablet for tablet ed9fd50fd4474d25883861f03d6333c0 (DEFAULT_TABLE table=rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:59.001466 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed9fd50fd4474d25883861f03d6333c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.003662 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 385630fa0b3d4fa7b48af2aa7852d2d5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:59.003655 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 385630fa0b3d4fa7b48af2aa7852d2d5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:59.004016 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385630fa0b3d4fa7b48af2aa7852d2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.004040 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385630fa0b3d4fa7b48af2aa7852d2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:59.011651 15242 tablet_bootstrap.cc:492] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:59.013255 15242 tablet_bootstrap.cc:654] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.018993 15243 tablet_bootstrap.cc:492] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:59.020886 15243 tablet_bootstrap.cc:654] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.029290 15243 tablet_bootstrap.cc:492] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:59.029469 15243 ts_tablet_manager.cc:1367] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240502 08:17:59.030273 15243 raft_consensus.cc:348] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:59.030596 15243 raft_consensus.cc:374] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.030736 15243 raft_consensus.cc:729] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.031824 15243 consensus_queue.cc:260] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:59.032572 15243 ts_tablet_manager.cc:1398] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:17:59.033416 15242 tablet_bootstrap.cc:492] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:59.033622 15242 ts_tablet_manager.cc:1367] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.008s
I20240502 08:17:59.033926 15243 tablet_bootstrap.cc:492] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:59.034409 15242 raft_consensus.cc:348] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:59.034741 15242 raft_consensus.cc:374] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.034881 15242 raft_consensus.cc:729] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.035169 15242 consensus_queue.cc:260] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:59.035794 15242 ts_tablet_manager.cc:1398] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:17:59.036105 15242 tablet_bootstrap.cc:492] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:17:59.036406 15245 tablet_bootstrap.cc:492] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:59.037851 15242 tablet_bootstrap.cc:654] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.038208 15243 tablet_bootstrap.cc:654] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.039029 15246 tablet_bootstrap.cc:492] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:17:59.043216 15245 tablet_bootstrap.cc:654] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.046237 15242 tablet_bootstrap.cc:492] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:17:59.046408 15242 ts_tablet_manager.cc:1367] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 08:17:59.046969 15246 tablet_bootstrap.cc:654] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.047041 15242 raft_consensus.cc:348] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:59.047259 15242 raft_consensus.cc:374] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.047407 15242 raft_consensus.cc:729] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.049139 15243 tablet_bootstrap.cc:492] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:59.049348 15243 ts_tablet_manager.cc:1367] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20240502 08:17:59.047683 15242 consensus_queue.cc:260] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:59.050037 15243 raft_consensus.cc:348] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:59.050292 15243 raft_consensus.cc:374] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.050410 15243 raft_consensus.cc:729] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.050443 15242 ts_tablet_manager.cc:1398] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:17:59.050709 15243 consensus_queue.cc:260] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:59.052325 15243 ts_tablet_manager.cc:1398] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:17:59.052592 15243 tablet_bootstrap.cc:492] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:17:59.053989 15243 tablet_bootstrap.cc:654] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.055428 15246 tablet_bootstrap.cc:492] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:17:59.055579 15246 ts_tablet_manager.cc:1367] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 08:17:59.056300 15246 raft_consensus.cc:348] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:59.056512 15246 raft_consensus.cc:374] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.056598 15246 raft_consensus.cc:729] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.056821 15246 consensus_queue.cc:260] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:59.057374 15246 ts_tablet_manager.cc:1398] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:17:59.061411 15245 tablet_bootstrap.cc:492] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:59.061549 15245 ts_tablet_manager.cc:1367] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240502 08:17:59.062240 15243 tablet_bootstrap.cc:492] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:17:59.062371 15243 ts_tablet_manager.cc:1367] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20240502 08:17:59.062311 15245 raft_consensus.cc:348] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:59.062508 15245 raft_consensus.cc:374] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.062593 15245 raft_consensus.cc:729] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.062800 15245 consensus_queue.cc:260] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:59.063102 15243 raft_consensus.cc:348] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:59.063270 15243 raft_consensus.cc:374] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.063294 15245 ts_tablet_manager.cc:1398] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:17:59.063352 15243 raft_consensus.cc:729] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.063560 15245 tablet_bootstrap.cc:492] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:59.063537 15243 consensus_queue.cc:260] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:59.064026 15243 ts_tablet_manager.cc:1398] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:17:59.065105 15245 tablet_bootstrap.cc:654] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.074209 15245 tablet_bootstrap.cc:492] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:59.074357 15245 ts_tablet_manager.cc:1367] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20240502 08:17:59.075073 15245 raft_consensus.cc:348] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:59.075296 15245 raft_consensus.cc:374] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.075364 15245 raft_consensus.cc:729] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.075596 15245 consensus_queue.cc:260] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:59.076123 15245 ts_tablet_manager.cc:1398] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:59.076589 15245 tablet_bootstrap.cc:492] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:17:59.077932 15245 tablet_bootstrap.cc:654] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:59.078003 14779 raft_consensus.cc:482] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:59.078162 14779 raft_consensus.cc:504] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:59.078833 14779 leader_election.cc:288] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:59.079743 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385630fa0b3d4fa7b48af2aa7852d2d5" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:59.079980 12990 raft_consensus.cc:2454] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:59.080236 12728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385630fa0b3d4fa7b48af2aa7852d2d5" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:17:59.080446 12728 raft_consensus.cc:2454] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:17:59.080817 12796 leader_election.cc:302] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:59.081104 14779 raft_consensus.cc:2781] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:59.081205 14779 raft_consensus.cc:482] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:59.081316 14779 raft_consensus.cc:3037] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.081589 15245 tablet_bootstrap.cc:492] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:17:59.081687 15245 ts_tablet_manager.cc:1367] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 08:17:59.082342 15245 raft_consensus.cc:348] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:59.082518 15245 raft_consensus.cc:374] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:59.082635 15245 raft_consensus.cc:729] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:17:59.082873 15245 consensus_queue.cc:260] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:59.083590 15245 ts_tablet_manager.cc:1398] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:17:59.083981 14779 raft_consensus.cc:504] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:59.084566 14779 leader_election.cc:288] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:17:59.084959 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385630fa0b3d4fa7b48af2aa7852d2d5" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:59.085094 12728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385630fa0b3d4fa7b48af2aa7852d2d5" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:17:59.085147 12990 raft_consensus.cc:3037] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.085245 12728 raft_consensus.cc:3037] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.087188 12990 raft_consensus.cc:2454] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:59.087322 12728 raft_consensus.cc:2454] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:17:59.087697 12796 leader_election.cc:302] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:59.087944 14779 raft_consensus.cc:2781] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:59.088086 14779 raft_consensus.cc:686] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:17:59.088300 14779 consensus_queue.cc:237] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:17:59.090890 12579 catalog_manager.cc:5484] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:17:59.129480 15178 raft_consensus.cc:482] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:59.129630 15178 raft_consensus.cc:504] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:59.130216 15178 leader_election.cc:288] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:59.130522 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1319f55da0b348ac8a325e7d04f0a11d" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:17:59.130722 12992 raft_consensus.cc:2454] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:59.130837 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1319f55da0b348ac8a325e7d04f0a11d" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:59.131060 12664 leader_election.cc:302] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:59.131088 12860 raft_consensus.cc:2454] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:17:59.131290 15178 raft_consensus.cc:2781] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:59.131410 15178 raft_consensus.cc:482] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:59.131501 15178 raft_consensus.cc:3037] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.133247 15178 raft_consensus.cc:504] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:59.133731 15178 leader_election.cc:288] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:17:59.134080 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1319f55da0b348ac8a325e7d04f0a11d" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:17:59.134122 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1319f55da0b348ac8a325e7d04f0a11d" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:59.134231 12992 raft_consensus.cc:3037] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.134286 12860 raft_consensus.cc:3037] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.136142 12992 raft_consensus.cc:2454] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:59.136175 12860 raft_consensus.cc:2454] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:17:59.136488 12664 leader_election.cc:302] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:17:59.136770 15178 raft_consensus.cc:2781] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:59.136915 15178 raft_consensus.cc:686] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:17:59.137394 15178 consensus_queue.cc:237] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:17:59.141796 12579 catalog_manager.cc:5484] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:17:59.171109 14595 raft_consensus.cc:482] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:59.171294 14595 raft_consensus.cc:504] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:59.171922 14595 leader_election.cc:288] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:59.172284 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9fd50fd4474d25883861f03d6333c0" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:17:59.172312 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9fd50fd4474d25883861f03d6333c0" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:17:59.172457 12860 raft_consensus.cc:2454] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:59.172506 13121 raft_consensus.cc:2454] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:17:59.172849 12926 leader_election.cc:302] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:17:59.173102 14595 raft_consensus.cc:2781] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:59.173182 14595 raft_consensus.cc:482] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:59.173277 14595 raft_consensus.cc:3037] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.175483 14595 raft_consensus.cc:504] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:59.176019 14595 leader_election.cc:288] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:17:59.176308 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9fd50fd4474d25883861f03d6333c0" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:17:59.176404 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed9fd50fd4474d25883861f03d6333c0" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:17:59.176501 12860 raft_consensus.cc:3037] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.176610 13121 raft_consensus.cc:3037] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:59.179260 12860 raft_consensus.cc:2454] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:59.179257 13121 raft_consensus.cc:2454] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:17:59.179579 12925 leader_election.cc:302] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:17:59.179783 14595 raft_consensus.cc:2781] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:59.179934 14595 raft_consensus.cc:686] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:17:59.180192 14595 consensus_queue.cc:237] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:17:59.183714 12579 catalog_manager.cc:5484] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:17:59.525470 14779 consensus_queue.cc:1035] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:59.530798 14779 consensus_queue.cc:1035] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:59.553881 15261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:59.554257 15261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:59.561899 15261 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:59.564873 14595 consensus_queue.cc:1035] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:59.568500 15259 consensus_queue.cc:1035] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_18'
I20240502 08:17:59.607196 15178 consensus_queue.cc:1035] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:59.613605 15069 consensus_queue.cc:1035] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 264.979 ms
time per row: 0.13249 ms
I20240502 08:18:00.002224 12586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:00.011308 13102 tablet_service.cc:1467] Processing CreateTablet for tablet ad53a92fbcc041378bb1958c65e997db (DEFAULT_TABLE table=rebalance_test_table_19 [id=c7115c9eea2a450b944a15086ffe02bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.011399 12841 tablet_service.cc:1467] Processing CreateTablet for tablet ad53a92fbcc041378bb1958c65e997db (DEFAULT_TABLE table=rebalance_test_table_19 [id=c7115c9eea2a450b944a15086ffe02bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.011799 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad53a92fbcc041378bb1958c65e997db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.011811 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad53a92fbcc041378bb1958c65e997db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.012306 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 581d2ed76e2e446eb84b94231ebb97f4 (DEFAULT_TABLE table=rebalance_test_table_19 [id=c7115c9eea2a450b944a15086ffe02bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.012625 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 581d2ed76e2e446eb84b94231ebb97f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.016386 15329 tablet_bootstrap.cc:492] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:00.016495 13100 tablet_service.cc:1467] Processing CreateTablet for tablet f1581261b4c144a9b670786eb3d26466 (DEFAULT_TABLE table=rebalance_test_table_19 [id=c7115c9eea2a450b944a15086ffe02bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.016841 13100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1581261b4c144a9b670786eb3d26466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.018599 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 581d2ed76e2e446eb84b94231ebb97f4 (DEFAULT_TABLE table=rebalance_test_table_19 [id=c7115c9eea2a450b944a15086ffe02bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.018980 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 581d2ed76e2e446eb84b94231ebb97f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.019032 12970 tablet_service.cc:1467] Processing CreateTablet for tablet f1581261b4c144a9b670786eb3d26466 (DEFAULT_TABLE table=rebalance_test_table_19 [id=c7115c9eea2a450b944a15086ffe02bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.019383 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1581261b4c144a9b670786eb3d26466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.019505 15329 tablet_bootstrap.cc:654] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.022749 12708 tablet_service.cc:1467] Processing CreateTablet for tablet ad53a92fbcc041378bb1958c65e997db (DEFAULT_TABLE table=rebalance_test_table_19 [id=c7115c9eea2a450b944a15086ffe02bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.023203 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad53a92fbcc041378bb1958c65e997db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.026839 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 581d2ed76e2e446eb84b94231ebb97f4 (DEFAULT_TABLE table=rebalance_test_table_19 [id=c7115c9eea2a450b944a15086ffe02bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.027190 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 581d2ed76e2e446eb84b94231ebb97f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.031113 12709 tablet_service.cc:1467] Processing CreateTablet for tablet f1581261b4c144a9b670786eb3d26466 (DEFAULT_TABLE table=rebalance_test_table_19 [id=c7115c9eea2a450b944a15086ffe02bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:00.031536 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1581261b4c144a9b670786eb3d26466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:00.035389 15330 tablet_bootstrap.cc:492] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:00.038296 15333 tablet_bootstrap.cc:492] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:00.041411 15332 tablet_bootstrap.cc:492] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:00.042160 15333 tablet_bootstrap.cc:654] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.044152 15330 tablet_bootstrap.cc:654] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.045871 15332 tablet_bootstrap.cc:654] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.048640 15330 tablet_bootstrap.cc:492] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:00.048858 15330 ts_tablet_manager.cc:1367] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 08:18:00.049702 15330 raft_consensus.cc:348] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:00.050033 15330 raft_consensus.cc:374] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.050159 15330 raft_consensus.cc:729] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.050436 15330 consensus_queue.cc:260] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:00.051126 15330 ts_tablet_manager.cc:1398] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:00.051357 15332 tablet_bootstrap.cc:492] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:00.051478 15332 ts_tablet_manager.cc:1367] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20240502 08:18:00.052048 15332 raft_consensus.cc:348] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:00.052506 15332 raft_consensus.cc:374] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.052651 15332 raft_consensus.cc:729] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.052973 15332 consensus_queue.cc:260] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:00.055023 15330 tablet_bootstrap.cc:492] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:00.056556 15330 tablet_bootstrap.cc:654] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.057327 15332 ts_tablet_manager.cc:1398] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 08:18:00.058593 15333 tablet_bootstrap.cc:492] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:00.059240 15333 ts_tablet_manager.cc:1367] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.008s
I20240502 08:18:00.060884 15329 tablet_bootstrap.cc:492] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:00.061080 15329 ts_tablet_manager.cc:1367] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.025s
I20240502 08:18:00.061944 15329 raft_consensus.cc:348] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:00.062201 15329 raft_consensus.cc:374] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.062319 15329 raft_consensus.cc:729] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.062625 15329 consensus_queue.cc:260] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:00.065742 15332 tablet_bootstrap.cc:492] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:00.066282 15329 ts_tablet_manager.cc:1398] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 08:18:00.068501 15332 tablet_bootstrap.cc:654] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.070498 15329 tablet_bootstrap.cc:492] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:00.072017 15329 tablet_bootstrap.cc:654] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.072585 15333 raft_consensus.cc:348] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:00.073347 15333 raft_consensus.cc:374] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.073536 15333 raft_consensus.cc:729] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.073935 15330 tablet_bootstrap.cc:492] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:00.074120 15330 ts_tablet_manager.cc:1367] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.012s
I20240502 08:18:00.073928 15333 consensus_queue.cc:260] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:00.074962 15333 ts_tablet_manager.cc:1398] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:18:00.074931 15330 raft_consensus.cc:348] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:00.075141 15330 raft_consensus.cc:374] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.075227 15330 raft_consensus.cc:729] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.075464 15330 consensus_queue.cc:260] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:00.076015 15330 ts_tablet_manager.cc:1398] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:00.076575 15330 tablet_bootstrap.cc:492] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:00.078104 15330 tablet_bootstrap.cc:654] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.080457 15333 tablet_bootstrap.cc:492] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:00.085315 15333 tablet_bootstrap.cc:654] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:00.088366 15329 tablet_bootstrap.cc:492] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:00.088569 15329 ts_tablet_manager.cc:1367] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20240502 08:18:00.089982 15329 raft_consensus.cc:348] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:00.090255 15329 raft_consensus.cc:374] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.090374 15329 raft_consensus.cc:729] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.090700 15329 consensus_queue.cc:260] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:00.094740 15329 ts_tablet_manager.cc:1398] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 08:18:00.099670 15330 tablet_bootstrap.cc:492] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:00.099839 15330 ts_tablet_manager.cc:1367] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.018s
I20240502 08:18:00.100138 15333 tablet_bootstrap.cc:492] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:00.100286 15333 ts_tablet_manager.cc:1367] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.004s
I20240502 08:18:00.100656 15330 raft_consensus.cc:348] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:00.100875 15330 raft_consensus.cc:374] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.100966 15330 raft_consensus.cc:729] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.101033 15333 raft_consensus.cc:348] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:00.101235 15333 raft_consensus.cc:374] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.101186 15330 consensus_queue.cc:260] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:00.101310 15333 raft_consensus.cc:729] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.101527 15333 consensus_queue.cc:260] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:00.101718 15330 ts_tablet_manager.cc:1398] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:00.102039 15333 ts_tablet_manager.cc:1398] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:00.106853 15332 tablet_bootstrap.cc:492] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:00.107019 15332 ts_tablet_manager.cc:1367] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.020s
I20240502 08:18:00.107664 15332 raft_consensus.cc:348] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:00.107883 15332 raft_consensus.cc:374] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:00.107968 15332 raft_consensus.cc:729] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:00.108188 15332 consensus_queue.cc:260] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:00.108685 15332 ts_tablet_manager.cc:1398] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:00.155267 15296 raft_consensus.cc:482] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:00.155426 15296 raft_consensus.cc:504] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:00.155994 15296 leader_election.cc:288] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:00.156344 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad53a92fbcc041378bb1958c65e997db" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:00.156477 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad53a92fbcc041378bb1958c65e997db" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:00.156625 12730 raft_consensus.cc:2454] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:00.156677 12859 raft_consensus.cc:2454] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:00.157063 13056 leader_election.cc:302] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:00.157306 15296 raft_consensus.cc:2781] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:00.157428 15296 raft_consensus.cc:482] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:00.157523 15296 raft_consensus.cc:3037] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.159621 15296 raft_consensus.cc:504] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:00.160128 15296 leader_election.cc:288] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:00.160508 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad53a92fbcc041378bb1958c65e997db" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:00.160635 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad53a92fbcc041378bb1958c65e997db" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:00.160717 12730 raft_consensus.cc:3037] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.160830 12859 raft_consensus.cc:3037] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.163473 12859 raft_consensus.cc:2454] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:00.163791 12730 raft_consensus.cc:2454] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:00.164172 13056 leader_election.cc:302] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:00.164453 15296 raft_consensus.cc:2781] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:00.164602 15296 raft_consensus.cc:686] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:18:00.164908 15296 consensus_queue.cc:237] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:00.168754 12584 catalog_manager.cc:5484] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:00.187924 15265 raft_consensus.cc:482] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:00.188068 15265 raft_consensus.cc:504] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:00.188791 15265 leader_election.cc:288] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:00.189105 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581d2ed76e2e446eb84b94231ebb97f4" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:00.189319 13121 raft_consensus.cc:2454] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:00.189261 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581d2ed76e2e446eb84b94231ebb97f4" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:00.189483 12990 raft_consensus.cc:2454] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:00.189735 12795 leader_election.cc:302] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:00.189992 15265 raft_consensus.cc:2781] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:00.190104 15265 raft_consensus.cc:482] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:00.190184 15265 raft_consensus.cc:3037] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.192714 15265 raft_consensus.cc:504] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:00.193011 15178 raft_consensus.cc:482] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:00.193297 15265 leader_election.cc:288] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:00.193163 15178 raft_consensus.cc:504] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:00.193617 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581d2ed76e2e446eb84b94231ebb97f4" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:00.193806 13121 raft_consensus.cc:3037] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.193887 15178 leader_election.cc:288] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:00.194207 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1581261b4c144a9b670786eb3d26466" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:00.194399 12990 raft_consensus.cc:2454] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:00.194437 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581d2ed76e2e446eb84b94231ebb97f4" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:00.194638 12992 raft_consensus.cc:3037] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.196442 13121 raft_consensus.cc:2454] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:00.196854 12795 leader_election.cc:302] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:00.197067 12992 raft_consensus.cc:2454] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:00.197144 15265 raft_consensus.cc:2781] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:00.197278 15265 raft_consensus.cc:686] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:18:00.197569 15265 consensus_queue.cc:237] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:00.197783 12664 leader_election.cc:302] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:18:00.199170 15178 raft_consensus.cc:2781] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:00.199316 15178 raft_consensus.cc:482] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:00.199415 15178 raft_consensus.cc:3037] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.200407 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1581261b4c144a9b670786eb3d26466" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:00.200628 13121 raft_consensus.cc:2454] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:00.202991 15178 raft_consensus.cc:504] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:00.203893 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1581261b4c144a9b670786eb3d26466" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:00.204119 12992 raft_consensus.cc:3037] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.205391 12584 catalog_manager.cc:5484] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:00.206841 12992 raft_consensus.cc:2454] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:00.207365 12664 leader_election.cc:302] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:18:00.207710 15319 raft_consensus.cc:2781] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:00.207826 15319 raft_consensus.cc:686] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:18:00.208204 15319 consensus_queue.cc:237] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:00.210117 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1581261b4c144a9b670786eb3d26466" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:00.210333 13121 raft_consensus.cc:3037] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:00.212311 12584 catalog_manager.cc:5484] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:00.212582 13121 raft_consensus.cc:2454] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:00.213071 15178 leader_election.cc:288] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
W20240502 08:18:00.595866 15345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:00.596143 15345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:18:00.604708 14779 consensus_queue.cc:1035] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:00.606698 15345 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:18:00.616832 15235 consensus_queue.cc:1035] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:00.628312 14779 consensus_queue.cc:1035] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:00.654615 15235 consensus_queue.cc:1035] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_19'
I20240502 08:18:00.690023 15319 consensus_queue.cc:1035] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:00.694842 15178 consensus_queue.cc:1035] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 213.775 ms
time per row: 0.106888 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 08:18:01.879709 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.5:44131
--local_ip_for_outbound_sockets=127.6.5.5
--tserver_master_addrs=127.6.5.62:46027
--webserver_port=37259
--webserver_interface=127.6.5.5
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:18:02.207083 15402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:02.207363 15402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:02.207509 15402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:02.215062 15402 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:18:02.215166 15402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:02.215318 15402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.5
I20240502 08:18:02.224598 15402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.5:44131
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.5.5
--webserver_port=37259
--tserver_master_addrs=127.6.5.62:46027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.5
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:02.225793 15402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:02.227540 15402 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:02.240262 15408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:02.240746 15409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:02.241245 15411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:02.241729 15402 server_base.cc:1034] running on GCE node
I20240502 08:18:02.242282 15402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:02.243194 15402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:02.244387 15402 hybrid_clock.cc:648] HybridClock initialized: now 1714637882244341 us; error 36 us; skew 500 ppm
I20240502 08:18:02.244719 15402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:02.250640 15402 webserver.cc:460] Webserver started at http://127.6.5.5:37259/ using document root <none> and password file <none>
I20240502 08:18:02.251379 15402 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:02.251505 15402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:02.256625 15402 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20240502 08:18:02.259346 15417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:02.260543 15402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 08:18:02.260674 15402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "db7893bb29a047b3b82cc60bfd8930a3"
format_stamp: "Formatted at 2024-05-02 08:17:27 on dist-test-slave-h7h6"
I20240502 08:18:02.261390 15402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:02.280252 15402 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:02.280958 15402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:02.281211 15402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:02.282063 15402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:02.283496 15402 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:02.283566 15402 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:02.283686 15402 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:02.283735 15402 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:02.313019 15402 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.5:44131
I20240502 08:18:02.313042 15532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.5:44131 every 8 connection(s)
I20240502 08:18:02.314389 15402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:18:02.318713 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 15402
I20240502 08:18:02.319301 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.6:44983
--local_ip_for_outbound_sockets=127.6.5.6
--tserver_master_addrs=127.6.5.62:46027
--webserver_port=45579
--webserver_interface=127.6.5.6
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:18:02.332427 15533 heartbeater.cc:347] Connected to a master server at 127.6.5.62:46027
I20240502 08:18:02.332732 15533 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:02.333585 15533 heartbeater.cc:511] Master 127.6.5.62:46027 requested a full tablet report, sending...
I20240502 08:18:02.335481 12586 ts_manager.cc:194] Re-registered known tserver with Master: db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:02.336634 12586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.5:46803
W20240502 08:18:02.661507 15537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:02.661761 15537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:02.661885 15537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:02.669211 15537 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:18:02.669332 15537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:02.669482 15537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.6
I20240502 08:18:02.678104 15537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.6:44983
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.5.6
--webserver_port=45579
--tserver_master_addrs=127.6.5.62:46027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.6
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:02.679286 15537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:02.680887 15537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:02.693423 15544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:02.694450 15537 server_base.cc:1034] running on GCE node
W20240502 08:18:02.694743 15546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:02.694849 15543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:02.695499 15537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:02.696372 15537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:02.697561 15537 hybrid_clock.cc:648] HybridClock initialized: now 1714637882697519 us; error 44 us; skew 500 ppm
I20240502 08:18:02.697846 15537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:02.699961 15537 webserver.cc:460] Webserver started at http://127.6.5.6:45579/ using document root <none> and password file <none>
I20240502 08:18:02.700654 15537 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:02.700781 15537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:02.706434 15537 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:18:02.709517 15552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:02.710952 15537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:18:02.711102 15537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
format_stamp: "Formatted at 2024-05-02 08:17:28 on dist-test-slave-h7h6"
I20240502 08:18:02.711915 15537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:02.746364 15537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:02.747249 15537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:02.747560 15537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:02.748636 15537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:02.750226 15537 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:02.750299 15537 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:02.750382 15537 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:02.750427 15537 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:02.794426 15537 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.6:44983
I20240502 08:18:02.794499 15666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.6:44983 every 8 connection(s)
I20240502 08:18:02.795827 15537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:18:02.803256 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 15537
I20240502 08:18:02.803862 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.7:38851
--local_ip_for_outbound_sockets=127.6.5.7
--tserver_master_addrs=127.6.5.62:46027
--webserver_port=41899
--webserver_interface=127.6.5.7
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:18:02.808549 15667 heartbeater.cc:347] Connected to a master server at 127.6.5.62:46027
I20240502 08:18:02.808862 15667 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:02.809648 15667 heartbeater.cc:511] Master 127.6.5.62:46027 requested a full tablet report, sending...
I20240502 08:18:02.812080 12586 ts_manager.cc:194] Re-registered known tserver with Master: 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:02.813038 12586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.6:43307
W20240502 08:18:03.134272 15670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:03.134564 15670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:03.134696 15670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:03.142093 15670 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:18:03.142194 15670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:03.142336 15670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.7
I20240502 08:18:03.151438 15670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.7:38851
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.5.7
--webserver_port=41899
--tserver_master_addrs=127.6.5.62:46027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.7
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:03.152606 15670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:03.154278 15670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:03.167475 15678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:03.167891 15677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:03.168315 15680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:03.168936 15670 server_base.cc:1034] running on GCE node
I20240502 08:18:03.169507 15670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:03.170387 15670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:03.171615 15670 hybrid_clock.cc:648] HybridClock initialized: now 1714637883171542 us; error 43 us; skew 500 ppm
I20240502 08:18:03.171993 15670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:03.174297 15670 webserver.cc:460] Webserver started at http://127.6.5.7:41899/ using document root <none> and password file <none>
I20240502 08:18:03.175127 15670 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:03.175259 15670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:03.181030 15670 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 08:18:03.184654 15686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:03.186199 15670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:03.186342 15670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9333cb10227b4801b4fa7c24e22e131b"
format_stamp: "Formatted at 2024-05-02 08:17:28 on dist-test-slave-h7h6"
I20240502 08:18:03.187372 15670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:03.229358 15670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:03.230103 15670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:03.230350 15670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:03.231295 15670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:03.232856 15670 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:03.232925 15670 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:03.233036 15670 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:03.233081 15670 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:03.271333 15670 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.7:38851
I20240502 08:18:03.271436 15799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.7:38851 every 8 connection(s)
I20240502 08:18:03.272882 15670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:18:03.276010 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 15670
I20240502 08:18:03.276711 6164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
/tmp/dist-test-taskTc4cwt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.5.8:46303
--local_ip_for_outbound_sockets=127.6.5.8
--tserver_master_addrs=127.6.5.62:46027
--webserver_port=38341
--webserver_interface=127.6.5.8
--builtin_ntp_servers=127.6.5.20:44879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:18:03.286043 15800 heartbeater.cc:347] Connected to a master server at 127.6.5.62:46027
I20240502 08:18:03.286335 15800 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:03.287182 15800 heartbeater.cc:511] Master 127.6.5.62:46027 requested a full tablet report, sending...
I20240502 08:18:03.288748 12586 ts_manager.cc:194] Re-registered known tserver with Master: 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:03.289520 12586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.7:59901
I20240502 08:18:03.339632 15533 heartbeater.cc:503] Master 127.6.5.62:46027 was elected leader, sending a full tablet report...
W20240502 08:18:03.607990 15803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:03.608227 15803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:03.608328 15803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:03.615931 15803 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:18:03.616050 15803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:03.616214 15803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.5.8
I20240502 08:18:03.624640 15803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.5.20:44879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.5.8:46303
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.5.8
--webserver_port=38341
--tserver_master_addrs=127.6.5.62:46027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.5.8
--log_dir=/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:03.625766 15803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:03.627606 15803 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:03.639371 15808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:03.639881 15811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:03.640321 15809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:03.640885 15803 server_base.cc:1034] running on GCE node
I20240502 08:18:03.641424 15803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:03.642374 15803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:03.643558 15803 hybrid_clock.cc:648] HybridClock initialized: now 1714637883643495 us; error 58 us; skew 500 ppm
I20240502 08:18:03.643862 15803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:03.646205 15803 webserver.cc:460] Webserver started at http://127.6.5.8:38341/ using document root <none> and password file <none>
I20240502 08:18:03.647099 15803 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:03.647238 15803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:03.652827 15803 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 08:18:03.655642 15817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:03.657409 15803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:18:03.657542 15803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "10f0e36203914b85b77e78609ffced98"
format_stamp: "Formatted at 2024-05-02 08:17:29 on dist-test-slave-h7h6"
I20240502 08:18:03.658257 15803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:03.686221 15803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:18:03.687059 15803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:03.687348 15803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:03.688297 15803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:03.689844 15803 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:03.689932 15803 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:03.690055 15803 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:03.690119 15803 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:18:03.723666 15803 rpc_server.cc:307] RPC server started. Bound to: 127.6.5.8:46303
I20240502 08:18:03.723726 15932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.5.8:46303 every 8 connection(s)
I20240502 08:18:03.725275 15803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:18:03.727511 6164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTc4cwt/build/asan/bin/kudu as pid 15803
I20240502 08:18:03.737152 12586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:03.740953 15933 heartbeater.cc:347] Connected to a master server at 127.6.5.62:46027
I20240502 08:18:03.741277 15933 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:03.742162 15933 heartbeater.cc:511] Master 127.6.5.62:46027 requested a full tablet report, sending...
I20240502 08:18:03.743862 12584 ts_manager.cc:194] Re-registered known tserver with Master: 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:03.744987 12584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.5.8:40305
I20240502 08:18:03.758744 12709 tablet_service.cc:1467] Processing CreateTablet for tablet a13a25e340fc4005a802157460d292e9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5a959630865149f198693dce1e1042ef]), partition=
I20240502 08:18:03.759186 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13a25e340fc4005a802157460d292e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.779613 15949 tablet_bootstrap.cc:492] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:03.783461 15949 tablet_bootstrap.cc:654] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.788236 15949 tablet_bootstrap.cc:492] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:03.788367 15949 ts_tablet_manager.cc:1367] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:18:03.789016 15949 raft_consensus.cc:348] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:03.789347 15949 raft_consensus.cc:374] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.789498 15949 raft_consensus.cc:729] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.792032 15949 consensus_queue.cc:260] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:03.792697 15949 ts_tablet_manager.cc:1398] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:03.803424 15735 tablet_service.cc:1467] Processing CreateTablet for tablet a13a25e340fc4005a802157460d292e9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5a959630865149f198693dce1e1042ef]), partition=
I20240502 08:18:03.804522 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13a25e340fc4005a802157460d292e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.806020 15602 tablet_service.cc:1467] Processing CreateTablet for tablet a13a25e340fc4005a802157460d292e9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5a959630865149f198693dce1e1042ef]), partition=
I20240502 08:18:03.806892 15602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13a25e340fc4005a802157460d292e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:03.815796 15667 heartbeater.cc:503] Master 127.6.5.62:46027 was elected leader, sending a full tablet report...
I20240502 08:18:03.819833 15951 tablet_bootstrap.cc:492] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:03.823318 15951 tablet_bootstrap.cc:654] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.824342 15951 log.cc:824] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:03.825712 15952 tablet_bootstrap.cc:492] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:03.826370 15951 tablet_bootstrap.cc:492] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:03.826614 15951 ts_tablet_manager.cc:1367] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240502 08:18:03.828804 15952 tablet_bootstrap.cc:654] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:03.831373 15952 log.cc:824] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:03.833171 15951 raft_consensus.cc:348] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:03.833537 15951 raft_consensus.cc:374] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.833688 15951 raft_consensus.cc:729] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.834378 15951 consensus_queue.cc:260] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:03.835049 15952 tablet_bootstrap.cc:492] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:03.835304 15952 ts_tablet_manager.cc:1367] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240502 08:18:03.843617 15951 ts_tablet_manager.cc:1398] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.017s user 0.005s sys 0.009s
W20240502 08:18:03.844785 12777 tablet.cc:2369] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:03.845690 15952 raft_consensus.cc:348] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:03.846076 15952 raft_consensus.cc:374] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:03.846549 15952 raft_consensus.cc:729] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:03.847297 15952 consensus_queue.cc:260] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:03.855011 15800 heartbeater.cc:503] Master 127.6.5.62:46027 was elected leader, sending a full tablet report...
I20240502 08:18:03.855619 15952 ts_tablet_manager.cc:1398] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.020s user 0.011s sys 0.007s
I20240502 08:18:03.861371 15955 raft_consensus.cc:482] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:03.861826 15955 raft_consensus.cc:504] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:03.863415 15955 leader_election.cc:288] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:03.869920 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13a25e340fc4005a802157460d292e9" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:03.870397 15755 raft_consensus.cc:2454] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:03.871112 15553 leader_election.cc:302] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:03.871420 15955 raft_consensus.cc:2781] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:03.871558 15955 raft_consensus.cc:482] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:03.871678 15955 raft_consensus.cc:3037] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.873761 15955 raft_consensus.cc:504] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:03.874331 15955 leader_election.cc:288] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:03.875038 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13a25e340fc4005a802157460d292e9" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:03.875205 15755 raft_consensus.cc:3037] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:03.877120 15755 raft_consensus.cc:2454] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:03.877524 15553 leader_election.cc:302] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:03.877871 15955 raft_consensus.cc:2781] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:03.882889 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13a25e340fc4005a802157460d292e9" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:03.883029 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13a25e340fc4005a802157460d292e9" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:03.883128 12727 raft_consensus.cc:2454] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:03.888629 15955 raft_consensus.cc:686] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:03.889259 15955 consensus_queue.cc:237] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:03.893544 12586 catalog_manager.cc:5484] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
W20240502 08:18:04.024885 15801 tablet.cc:2369] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:04.026930 12586 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:04.038911 12730 raft_consensus.cc:3037] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.039659 15755 raft_consensus.cc:1261] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:04.040648 15955 consensus_queue.cc:1035] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:04.041538 12730 raft_consensus.cc:1261] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:04.042606 15955 consensus_queue.cc:1035] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:04.048134 15668 tablet.cc:2369] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:04.056106 12584 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet a13a25e340fc4005a802157460d292e9 (table rebalancer_dynamic_table [id=5a959630865149f198693dce1e1042ef]). Expected version 1 got 0
I20240502 08:18:04.059336 15962 mvcc.cc:205] Tried to move back new op lower bound from 7023156773013004288 to 7023156772414627840. Current Snapshot: MvccSnapshot[applied={T|T < 7023156773013004288}]
I20240502 08:18:04.059494 12586 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet a13a25e340fc4005a802157460d292e9 (table rebalancer_dynamic_table [id=5a959630865149f198693dce1e1042ef]). Expected version 1 got 0
I20240502 08:18:04.062685 15970 tablet.cc:1722] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:04.064096 12584 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet a13a25e340fc4005a802157460d292e9 (table rebalancer_dynamic_table [id=5a959630865149f198693dce1e1042ef]). Expected version 1 got 0
I20240502 08:18:04.064322 15970 tablet.cc:1620] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:04.070097 15967 tablet.cc:1722] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:04.072252 15967 tablet.cc:1620] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:04.074599 12586 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=5a959630865149f198693dce1e1042ef] alter complete (version 1)
I20240502 08:18:04.075317 15969 tablet.cc:1722] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:04.078217 12586 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet a13a25e340fc4005a802157460d292e9 (table rebalancer_dynamic_table [id=5a959630865149f198693dce1e1042ef]). Expected version 1 got 0
I20240502 08:18:04.081873 15969 tablet.cc:1620] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:04.184957 12586 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:04.193940 15967 tablet.cc:1722] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:04.194413 15967 tablet.cc:1620] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:04.194967 15970 tablet.cc:1722] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:04.195405 15970 tablet.cc:1620] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:04.203867 15969 tablet.cc:1722] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:04.204325 15969 tablet.cc:1620] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:04.207091 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=5a959630865149f198693dce1e1042ef] alter complete (version 2)
I20240502 08:18:04.229794 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 08:18:04.229722 12586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:04.229992 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 08:18:04.234987 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 4dffea0ca23d44c195472dbe603a4ff6
I20240502 08:18:04.235752 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet c0edac44fda6413d9d69d64d929532d6
I20240502 08:18:04.236001 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dffea0ca23d44c195472dbe603a4ff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:04.236153 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet cd8ddfa7c7a34b0c849e498ff94f8046
I20240502 08:18:04.236202 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dffea0ca23d44c195472dbe603a4ff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:04.236562 12972 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dffea0ca23d44c195472dbe603a4ff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:04.236980 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet cd8ddfa7c7a34b0c849e498ff94f8046 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:04.237313 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet c0edac44fda6413d9d69d64d929532d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:04.237509 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet c0edac44fda6413d9d69d64d929532d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:04.238008 12840 tablet_service.cc:1514] Processing DeleteTablet for tablet cd8ddfa7c7a34b0c849e498ff94f8046 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:04.238355 15993 tablet_replica.cc:330] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:04.238942 12708 tablet_service.cc:1514] Processing DeleteTablet for tablet c0edac44fda6413d9d69d64d929532d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:04.239410 12710 tablet_service.cc:1514] Processing DeleteTablet for tablet cd8ddfa7c7a34b0c849e498ff94f8046 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:04.240321 15993 raft_consensus.cc:2229] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.240669 15993 raft_consensus.cc:2258] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.242130 15993 ts_tablet_manager.cc:1875] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.246840 15994 tablet_replica.cc:330] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:04.247320 15994 raft_consensus.cc:2229] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.247671 15994 raft_consensus.cc:2258] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.248996 15994 ts_tablet_manager.cc:1875] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.252924 15993 ts_tablet_manager.cc:1888] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.252970 15996 tablet_replica.cc:330] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:04.253082 15993 log.cc:1196] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/4dffea0ca23d44c195472dbe603a4ff6
I20240502 08:18:04.253494 15996 raft_consensus.cc:2229] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:04.253574 15993 ts_tablet_manager.cc:1909] T 4dffea0ca23d44c195472dbe603a4ff6 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:04.253931 15996 raft_consensus.cc:2258] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.254982 15996 ts_tablet_manager.cc:1875] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.258337 15995 tablet_replica.cc:330] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:04.258754 15995 raft_consensus.cc:2229] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:04.259088 15994 ts_tablet_manager.cc:1888] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.259233 15994 log.cc:1196] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/4dffea0ca23d44c195472dbe603a4ff6
I20240502 08:18:04.259300 15995 raft_consensus.cc:2258] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.259443 15996 ts_tablet_manager.cc:1888] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.259580 15996 log.cc:1196] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/c0edac44fda6413d9d69d64d929532d6
I20240502 08:18:04.259740 15994 ts_tablet_manager.cc:1909] T 4dffea0ca23d44c195472dbe603a4ff6 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:04.260169 15996 ts_tablet_manager.cc:1909] T c0edac44fda6413d9d69d64d929532d6 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:04.260910 15995 ts_tablet_manager.cc:1875] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.261236 15994 tablet_replica.cc:330] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:04.261315 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 4dffea0ca23d44c195472dbe603a4ff6 (table rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]) successfully deleted
I20240502 08:18:04.261430 15994 raft_consensus.cc:2229] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.261631 15994 raft_consensus.cc:2258] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.261670 15996 tablet_replica.cc:330] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:04.261679 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet c0edac44fda6413d9d69d64d929532d6 (table rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]) successfully deleted
I20240502 08:18:04.261845 15996 raft_consensus.cc:2229] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.262053 15996 raft_consensus.cc:2258] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.262252 15994 ts_tablet_manager.cc:1875] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.262684 15996 ts_tablet_manager.cc:1875] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.266922 15996 ts_tablet_manager.cc:1888] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.267014 15993 tablet_replica.cc:330] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:04.267083 15996 log.cc:1196] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/cd8ddfa7c7a34b0c849e498ff94f8046
I20240502 08:18:04.267094 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 4dffea0ca23d44c195472dbe603a4ff6 (table rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]) successfully deleted
I20240502 08:18:04.267400 15993 raft_consensus.cc:2229] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:04.267493 15996 ts_tablet_manager.cc:1909] T cd8ddfa7c7a34b0c849e498ff94f8046 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:04.267832 15993 raft_consensus.cc:2258] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.268285 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet cd8ddfa7c7a34b0c849e498ff94f8046 (table rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]) successfully deleted
I20240502 08:18:04.268483 15993 ts_tablet_manager.cc:1875] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.274178 15993 ts_tablet_manager.cc:1888] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.274363 15993 log.cc:1196] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/cd8ddfa7c7a34b0c849e498ff94f8046
I20240502 08:18:04.274802 15993 ts_tablet_manager.cc:1909] T cd8ddfa7c7a34b0c849e498ff94f8046 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:04.275859 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet cd8ddfa7c7a34b0c849e498ff94f8046 (table rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]) successfully deleted
I20240502 08:18:04.278363 15994 ts_tablet_manager.cc:1888] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.278584 15994 log.cc:1196] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/c0edac44fda6413d9d69d64d929532d6
I20240502 08:18:04.279074 15994 ts_tablet_manager.cc:1909] T c0edac44fda6413d9d69d64d929532d6 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:04.279843 15995 ts_tablet_manager.cc:1888] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.279944 15994 tablet_replica.cc:330] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:04.280063 15995 log.cc:1196] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/4dffea0ca23d44c195472dbe603a4ff6
I20240502 08:18:04.280143 15994 raft_consensus.cc:2229] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.280375 15994 raft_consensus.cc:2258] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.280648 15995 ts_tablet_manager.cc:1909] T 4dffea0ca23d44c195472dbe603a4ff6 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:04.280982 15994 ts_tablet_manager.cc:1875] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.281687 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet c0edac44fda6413d9d69d64d929532d6 (table rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]) successfully deleted
I20240502 08:18:04.284214 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 4dffea0ca23d44c195472dbe603a4ff6 (table rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]) successfully deleted
I20240502 08:18:04.285006 15995 tablet_replica.cc:330] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:04.285262 15995 raft_consensus.cc:2229] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.285523 15995 raft_consensus.cc:2258] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.286193 15995 ts_tablet_manager.cc:1875] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.286294 15994 ts_tablet_manager.cc:1888] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.286406 15994 log.cc:1196] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/cd8ddfa7c7a34b0c849e498ff94f8046
I20240502 08:18:04.286926 15994 ts_tablet_manager.cc:1909] T cd8ddfa7c7a34b0c849e498ff94f8046 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:04.287140 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 495ea7de81d34b50a92f6cb52795fadf (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=57f8b963d2e24f929fa5234b3c552ab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.287683 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495ea7de81d34b50a92f6cb52795fadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.291829 15995 ts_tablet_manager.cc:1888] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.291986 15995 log.cc:1196] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/c0edac44fda6413d9d69d64d929532d6
I20240502 08:18:04.291904 15600 tablet_service.cc:1467] Processing CreateTablet for tablet 566ef75f039f41f18b76c2e4f3e640dd (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=57f8b963d2e24f929fa5234b3c552ab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.292306 15995 ts_tablet_manager.cc:1909] T c0edac44fda6413d9d69d64d929532d6 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:04.292315 15600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566ef75f039f41f18b76c2e4f3e640dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.295343 15602 tablet_service.cc:1467] Processing CreateTablet for tablet 9e51af7112424e52a9e9c3200a30f86d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=57f8b963d2e24f929fa5234b3c552ab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.295652 15602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e51af7112424e52a9e9c3200a30f86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.296525 15951 tablet_bootstrap.cc:492] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:04.297825 15951 tablet_bootstrap.cc:654] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.298702 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 495ea7de81d34b50a92f6cb52795fadf (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=57f8b963d2e24f929fa5234b3c552ab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.299163 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495ea7de81d34b50a92f6cb52795fadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.306660 16002 tablet_bootstrap.cc:492] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:04.309777 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet c0edac44fda6413d9d69d64d929532d6 (table rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]) successfully deleted
I20240502 08:18:04.310129 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet cd8ddfa7c7a34b0c849e498ff94f8046 (table rebalance_test_table_0 [id=a69c252ec2344b58ba3092614beb615d]) successfully deleted
I20240502 08:18:04.310148 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 566ef75f039f41f18b76c2e4f3e640dd (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=57f8b963d2e24f929fa5234b3c552ab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.310592 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566ef75f039f41f18b76c2e4f3e640dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.312021 15951 tablet_bootstrap.cc:492] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:04.312127 15951 ts_tablet_manager.cc:1367] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.003s
I20240502 08:18:04.313051 15951 raft_consensus.cc:348] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.313231 15951 raft_consensus.cc:374] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.313309 15951 raft_consensus.cc:729] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.313505 15951 consensus_queue.cc:260] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.317428 15951 ts_tablet_manager.cc:1398] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240502 08:18:04.318097 15951 tablet_bootstrap.cc:492] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:04.319581 15951 tablet_bootstrap.cc:654] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.321442 15951 tablet_bootstrap.cc:492] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:04.321601 15951 ts_tablet_manager.cc:1367] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:04.321568 12586 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:04.322242 15951 raft_consensus.cc:348] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.322497 15951 raft_consensus.cc:374] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.322582 15951 raft_consensus.cc:729] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.322825 15951 consensus_queue.cc:260] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.324148 15951 ts_tablet_manager.cc:1398] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:04.324431 15951 tablet_bootstrap.cc:492] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:04.326069 15955 raft_consensus.cc:482] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.326227 15955 raft_consensus.cc:504] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.326985 15955 leader_election.cc:288] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:04.334086 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 566ef75f039f41f18b76c2e4f3e640dd (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=57f8b963d2e24f929fa5234b3c552ab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.334748 16005 tablet_bootstrap.cc:492] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:04.334808 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566ef75f039f41f18b76c2e4f3e640dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.336480 16005 tablet_bootstrap.cc:654] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.338933 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 9e51af7112424e52a9e9c3200a30f86d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=57f8b963d2e24f929fa5234b3c552ab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.339303 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e51af7112424e52a9e9c3200a30f86d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:18:04.339319 15553 leader_election.cc:341] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Not found: Tablet not found: 566ef75f039f41f18b76c2e4f3e640dd
I20240502 08:18:04.340443 15967 tablet.cc:1722] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:04.347409 15952 tablet_bootstrap.cc:492] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:04.348886 15970 tablet.cc:1722] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:04.349017 15952 tablet_bootstrap.cc:654] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.359269 16002 tablet_bootstrap.cc:654] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.362779 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=5a959630865149f198693dce1e1042ef] alter complete (version 3)
I20240502 08:18:04.364444 16002 tablet_bootstrap.cc:492] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:04.364640 16002 ts_tablet_manager.cc:1367] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20240502 08:18:04.364230 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 495ea7de81d34b50a92f6cb52795fadf (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=57f8b963d2e24f929fa5234b3c552ab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.365473 15952 tablet_bootstrap.cc:492] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:04.365432 16002 raft_consensus.cc:348] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.365636 16002 raft_consensus.cc:374] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.365602 15952 ts_tablet_manager.cc:1367] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.004s
I20240502 08:18:04.365731 16002 raft_consensus.cc:729] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.365960 16002 consensus_queue.cc:260] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.366487 16002 ts_tablet_manager.cc:1398] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:04.366612 15952 raft_consensus.cc:348] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.367918 15952 raft_consensus.cc:374] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.368021 15952 raft_consensus.cc:729] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.368225 15952 consensus_queue.cc:260] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.368752 15952 ts_tablet_manager.cc:1398] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:04.369060 15952 tablet_bootstrap.cc:492] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:04.369665 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495ea7de81d34b50a92f6cb52795fadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.370752 15952 tablet_bootstrap.cc:654] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.358273 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 9e51af7112424e52a9e9c3200a30f86d (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=57f8b963d2e24f929fa5234b3c552ab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:04.371982 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e51af7112424e52a9e9c3200a30f86d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.378633 15952 tablet_bootstrap.cc:492] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:04.378800 15952 ts_tablet_manager.cc:1367] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 08:18:04.379534 15952 raft_consensus.cc:348] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.379828 15952 raft_consensus.cc:374] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.379925 15952 raft_consensus.cc:729] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.380254 15952 consensus_queue.cc:260] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.383971 15952 ts_tablet_manager.cc:1398] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:04.398787 15951 tablet_bootstrap.cc:654] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.412382 15969 tablet.cc:1722] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:04.426707 15951 tablet_bootstrap.cc:492] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:04.426913 15951 ts_tablet_manager.cc:1367] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.103s user 0.003s sys 0.000s
I20240502 08:18:04.427906 15951 raft_consensus.cc:348] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.428133 15951 raft_consensus.cc:374] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.428225 15951 raft_consensus.cc:729] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.428454 15951 consensus_queue.cc:260] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.429093 15951 ts_tablet_manager.cc:1398] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:04.436683 12991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566ef75f039f41f18b76c2e4f3e640dd" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
W20240502 08:18:04.437660 15554 leader_election.cc:341] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:04.437804 15554 leader_election.cc:302] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: 60b6e265e7754b7187307d05ef405dd4, 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:04.438143 15960 raft_consensus.cc:2726] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:04.441833 16016 tablet_bootstrap.cc:492] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:04.443085 16005 tablet_bootstrap.cc:492] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:04.443238 16005 ts_tablet_manager.cc:1367] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.110s user 0.021s sys 0.047s
I20240502 08:18:04.444363 16005 raft_consensus.cc:348] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.444579 16005 raft_consensus.cc:374] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.444679 16005 raft_consensus.cc:729] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.444953 16005 consensus_queue.cc:260] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.445621 16005 ts_tablet_manager.cc:1398] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 08:18:04.446015 16016 tablet_bootstrap.cc:654] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.447688 16016 log.cc:824] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:04.452874 16019 tablet_bootstrap.cc:492] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:04.457494 16019 tablet_bootstrap.cc:654] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.458794 16019 log.cc:824] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:04.459441 16016 tablet_bootstrap.cc:492] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:04.459740 16016 ts_tablet_manager.cc:1367] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.003s
I20240502 08:18:04.465759 16019 tablet_bootstrap.cc:492] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:04.466058 16019 ts_tablet_manager.cc:1367] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240502 08:18:04.469254 12586 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:04.469496 12586 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:04.469307 16016 raft_consensus.cc:348] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.469744 16016 raft_consensus.cc:374] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.469856 16016 raft_consensus.cc:729] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.470672 16016 consensus_queue.cc:260] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.475322 16019 raft_consensus.cc:348] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.475768 16019 raft_consensus.cc:374] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.475911 16019 raft_consensus.cc:729] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.476563 16019 consensus_queue.cc:260] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.478080 12586 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet a13a25e340fc4005a802157460d292e9
I20240502 08:18:04.479302 15601 tablet_service.cc:1514] Processing DeleteTablet for tablet a13a25e340fc4005a802157460d292e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:04.480191 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet a13a25e340fc4005a802157460d292e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:04.481205 15734 tablet_service.cc:1514] Processing DeleteTablet for tablet a13a25e340fc4005a802157460d292e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:04.482609 15994 tablet_replica.cc:330] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:04.482851 15994 raft_consensus.cc:2229] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.483052 15994 raft_consensus.cc:2258] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.484266 15994 ts_tablet_manager.cc:1875] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.490336 16016 ts_tablet_manager.cc:1398] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.030s user 0.018s sys 0.000s
I20240502 08:18:04.491068 16026 tablet_replica.cc:330] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:04.491564 16026 raft_consensus.cc:2229] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:04.492205 16026 raft_consensus.cc:2258] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.492863 16027 tablet_replica.cc:330] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:04.493232 15933 heartbeater.cc:503] Master 127.6.5.62:46027 was elected leader, sending a full tablet report...
I20240502 08:18:04.493384 16027 raft_consensus.cc:2229] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.493773 16027 raft_consensus.cc:2258] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.495061 15994 ts_tablet_manager.cc:1888] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:04.495221 15994 log.cc:1196] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/a13a25e340fc4005a802157460d292e9
I20240502 08:18:04.495630 15994 ts_tablet_manager.cc:1909] T a13a25e340fc4005a802157460d292e9 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:04.496598 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet a13a25e340fc4005a802157460d292e9 (table rebalancer_dynamic_table_new_name [id=5a959630865149f198693dce1e1042ef]) successfully deleted
I20240502 08:18:04.499433 16027 ts_tablet_manager.cc:1875] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.500463 16026 ts_tablet_manager.cc:1875] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.510893 16019 ts_tablet_manager.cc:1398] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.045s user 0.020s sys 0.022s
I20240502 08:18:04.512184 16026 ts_tablet_manager.cc:1888] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:04.512353 16026 log.cc:1196] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/a13a25e340fc4005a802157460d292e9
I20240502 08:18:04.512916 16026 ts_tablet_manager.cc:1909] T a13a25e340fc4005a802157460d292e9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting consensus metadata
I20240502 08:18:04.514640 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet a13a25e340fc4005a802157460d292e9 (table rebalancer_dynamic_table_new_name [id=5a959630865149f198693dce1e1042ef]) successfully deleted
I20240502 08:18:04.515359 16027 ts_tablet_manager.cc:1888] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:04.515514 16027 log.cc:1196] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/a13a25e340fc4005a802157460d292e9
I20240502 08:18:04.516067 16027 ts_tablet_manager.cc:1909] T a13a25e340fc4005a802157460d292e9 P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:04.517828 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet a13a25e340fc4005a802157460d292e9 (table rebalancer_dynamic_table_new_name [id=5a959630865149f198693dce1e1042ef]) successfully deleted
I20240502 08:18:04.523489 15960 raft_consensus.cc:482] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.523671 15960 raft_consensus.cc:504] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.524590 15960 leader_election.cc:288] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:04.525506 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e51af7112424e52a9e9c3200a30f86d" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:04.525730 15755 raft_consensus.cc:2454] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:04.526309 15553 leader_election.cc:302] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:04.526625 15960 raft_consensus.cc:2781] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:04.526718 15960 raft_consensus.cc:482] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:04.526829 15960 raft_consensus.cc:3037] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:18:04.528313 12908 tablet.cc:2369] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:04.530565 15960 raft_consensus.cc:504] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.532075 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e51af7112424e52a9e9c3200a30f86d" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:04.532128 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e51af7112424e52a9e9c3200a30f86d" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:04.532578 15886 raft_consensus.cc:3037] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.535491 15886 raft_consensus.cc:2454] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:04.535549 15960 leader_election.cc:288] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:04.536098 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e51af7112424e52a9e9c3200a30f86d" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:04.536298 15755 raft_consensus.cc:3037] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.536609 15555 leader_election.cc:302] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:04.536893 15960 raft_consensus.cc:2781] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:04.537124 15960 raft_consensus.cc:686] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:04.537375 15960 consensus_queue.cc:237] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.539237 15755 raft_consensus.cc:2454] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:04.544672 12579 catalog_manager.cc:5484] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
W20240502 08:18:04.567055 15534 tablet.cc:2369] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:04.581696 12583 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:04.590265 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 93bd16eac6764d0bb5e1389ba990ff4b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d8c94f188c34464b68f2904c54b4712]), partition=
I20240502 08:18:04.590646 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bd16eac6764d0bb5e1389ba990ff4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.590968 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 93bd16eac6764d0bb5e1389ba990ff4b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d8c94f188c34464b68f2904c54b4712]), partition=
I20240502 08:18:04.591252 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bd16eac6764d0bb5e1389ba990ff4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.595067 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 93bd16eac6764d0bb5e1389ba990ff4b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d8c94f188c34464b68f2904c54b4712]), partition=
I20240502 08:18:04.595422 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bd16eac6764d0bb5e1389ba990ff4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:04.596874 15951 tablet_bootstrap.cc:492] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:04.598345 15951 tablet_bootstrap.cc:654] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.600102 15951 tablet_bootstrap.cc:492] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:04.600257 15951 ts_tablet_manager.cc:1367] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:18:04.601016 15951 raft_consensus.cc:348] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.601248 15951 raft_consensus.cc:374] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.601334 15951 raft_consensus.cc:729] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.601588 15951 consensus_queue.cc:260] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.604517 15951 ts_tablet_manager.cc:1398] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:04.604902 16019 tablet_bootstrap.cc:492] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:04.606302 16019 tablet_bootstrap.cc:654] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.609340 16016 tablet_bootstrap.cc:492] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:04.610908 16016 tablet_bootstrap.cc:654] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:04.613817 16016 tablet_bootstrap.cc:492] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:04.613965 16016 ts_tablet_manager.cc:1367] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:04.614909 16016 raft_consensus.cc:348] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.615119 16016 raft_consensus.cc:374] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.615221 16016 raft_consensus.cc:729] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.615427 16019 tablet_bootstrap.cc:492] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:04.615463 16016 consensus_queue.cc:260] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.616091 15960 raft_consensus.cc:482] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.616216 15960 raft_consensus.cc:504] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.616331 16019 ts_tablet_manager.cc:1367] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240502 08:18:04.616925 15960 leader_election.cc:288] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:04.617339 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bd16eac6764d0bb5e1389ba990ff4b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:04.617547 15886 raft_consensus.cc:2454] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:04.617961 15555 leader_election.cc:302] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:04.618382 15960 raft_consensus.cc:2781] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:04.618511 15960 raft_consensus.cc:482] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:04.618597 15960 raft_consensus.cc:3037] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.619087 16025 raft_consensus.cc:482] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.619292 16025 raft_consensus.cc:504] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.621179 15960 raft_consensus.cc:504] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.621237 16025 leader_election.cc:288] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:04.622153 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bd16eac6764d0bb5e1389ba990ff4b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:04.622337 15886 raft_consensus.cc:3037] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.625620 15886 raft_consensus.cc:2454] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:04.626945 15555 leader_election.cc:302] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:04.627208 16008 raft_consensus.cc:2781] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:04.627329 16008 raft_consensus.cc:686] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:04.627640 16008 consensus_queue.cc:237] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.629037 16010 raft_consensus.cc:482] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.629204 16010 raft_consensus.cc:504] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.630066 16010 leader_election.cc:288] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:04.635123 15960 leader_election.cc:288] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:04.635589 16016 ts_tablet_manager.cc:1398] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 08:18:04.643388 15960 raft_consensus.cc:482] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.643549 15960 raft_consensus.cc:504] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.644692 15960 leader_election.cc:288] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:04.644754 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495ea7de81d34b50a92f6cb52795fadf" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:04.645184 15488 raft_consensus.cc:2454] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:04.645745 12796 leader_election.cc:302] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db7893bb29a047b3b82cc60bfd8930a3, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:04.646018 16010 raft_consensus.cc:2781] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:04.646133 16010 raft_consensus.cc:482] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:04.646219 16010 raft_consensus.cc:3037] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.646656 12579 catalog_manager.cc:5484] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:04.648732 16010 raft_consensus.cc:504] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.649278 16010 leader_election.cc:288] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:04.652472 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495ea7de81d34b50a92f6cb52795fadf" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:04.652662 15488 raft_consensus.cc:3037] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.654925 15488 raft_consensus.cc:2454] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:04.655295 12796 leader_election.cc:302] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db7893bb29a047b3b82cc60bfd8930a3, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:04.655591 16010 raft_consensus.cc:2781] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:04.655731 16010 raft_consensus.cc:686] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:18:04.656009 16010 consensus_queue.cc:237] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.659145 16019 raft_consensus.cc:348] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.659417 16019 raft_consensus.cc:374] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:04.659579 16019 raft_consensus.cc:729] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:04.659883 16019 consensus_queue.cc:260] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:04.661108 16019 ts_tablet_manager.cc:1398] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20240502 08:18:04.673168 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bd16eac6764d0bb5e1389ba990ff4b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:04.673415 15488 raft_consensus.cc:2454] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:04.676551 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bd16eac6764d0bb5e1389ba990ff4b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:04.676800 15488 raft_consensus.cc:3037] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.679914 15488 raft_consensus.cc:2454] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:04.680472 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495ea7de81d34b50a92f6cb52795fadf" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:04.680806 15488 raft_consensus.cc:2379] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in current term 1: Already voted for candidate e1dd14f7ca764a1490df419f13f2804a in this term.
I20240502 08:18:04.686259 12579 catalog_manager.cc:5484] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
W20240502 08:18:04.686575 13038 tablet.cc:2369] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:04.703796 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495ea7de81d34b50a92f6cb52795fadf" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:04.704195 15621 raft_consensus.cc:3037] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.704383 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495ea7de81d34b50a92f6cb52795fadf" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:04.706557 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495ea7de81d34b50a92f6cb52795fadf" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:04.706897 15621 raft_consensus.cc:2454] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:04.710569 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495ea7de81d34b50a92f6cb52795fadf" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:04.710762 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "495ea7de81d34b50a92f6cb52795fadf" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:04.711159 15554 leader_election.cc:302] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: db7893bb29a047b3b82cc60bfd8930a3, e1dd14f7ca764a1490df419f13f2804a
I20240502 08:18:04.712647 15621 raft_consensus.cc:2379] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db7893bb29a047b3b82cc60bfd8930a3 in current term 1: Already voted for candidate e1dd14f7ca764a1490df419f13f2804a in this term.
I20240502 08:18:04.712783 15960 raft_consensus.cc:2726] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:04.713793 15419 leader_election.cc:302] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db7893bb29a047b3b82cc60bfd8930a3; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a
I20240502 08:18:04.714337 16025 raft_consensus.cc:2726] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 08:18:04.727617 15934 tablet.cc:2369] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:04.732599 15956 raft_consensus.cc:482] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:04.732741 15956 raft_consensus.cc:504] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.734146 15956 leader_election.cc:288] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:04.738088 12579 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 08:18:04.738350 12579 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 08:18:04.741868 12991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566ef75f039f41f18b76c2e4f3e640dd" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:04.742074 12991 raft_consensus.cc:2454] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:04.742347 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 17bd47b83e7f42698a2071cf82c6aa94
I20240502 08:18:04.742583 15688 leader_election.cc:302] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:04.742802 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566ef75f039f41f18b76c2e4f3e640dd" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:04.742956 15956 raft_consensus.cc:2781] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:04.743039 15956 raft_consensus.cc:482] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:04.743021 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 17bd47b83e7f42698a2071cf82c6aa94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:04.743029 15621 raft_consensus.cc:2454] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:04.743094 15956 raft_consensus.cc:3037] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.743289 15996 tablet_replica.cc:330] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:04.743489 15996 raft_consensus.cc:2229] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.743734 15996 raft_consensus.cc:2258] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.744313 15996 ts_tablet_manager.cc:1875] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.745123 15956 raft_consensus.cc:504] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.745571 15956 leader_election.cc:288] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:04.746107 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566ef75f039f41f18b76c2e4f3e640dd" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:04.746305 15621 raft_consensus.cc:3037] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.746448 12991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566ef75f039f41f18b76c2e4f3e640dd" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:04.746680 12991 raft_consensus.cc:3037] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:04.748602 12991 raft_consensus.cc:2454] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:04.748936 15688 leader_election.cc:302] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:04.749114 15621 raft_consensus.cc:2454] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:04.749166 15956 raft_consensus.cc:2781] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:04.749588 15996 ts_tablet_manager.cc:1888] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.749711 15996 log.cc:1196] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/17bd47b83e7f42698a2071cf82c6aa94
I20240502 08:18:04.750057 15996 ts_tablet_manager.cc:1909] T 17bd47b83e7f42698a2071cf82c6aa94 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:04.750886 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 97d0ce51e8ad46dda59ae9d14b11ac1a
I20240502 08:18:04.751204 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet 17bd47b83e7f42698a2071cf82c6aa94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:04.751329 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet acf6ca36b79e4c69a61892d51978baa5
I20240502 08:18:04.751503 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 17bd47b83e7f42698a2071cf82c6aa94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:04.751780 15993 tablet_replica.cc:330] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:04.751788 15994 tablet_replica.cc:330] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:04.751972 15994 raft_consensus.cc:2229] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:04.751968 15993 raft_consensus.cc:2229] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.752224 15993 raft_consensus.cc:2258] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.752453 15994 raft_consensus.cc:2258] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.752779 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet acf6ca36b79e4c69a61892d51978baa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:04.752895 15993 ts_tablet_manager.cc:1875] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.753057 15994 ts_tablet_manager.cc:1875] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.753175 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 17bd47b83e7f42698a2071cf82c6aa94 (table rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]) successfully deleted
I20240502 08:18:04.753355 15996 tablet_replica.cc:330] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:04.753527 15996 raft_consensus.cc:2229] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:04.753790 15996 raft_consensus.cc:2258] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.754292 15996 ts_tablet_manager.cc:1875] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.754633 12972 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d0ce51e8ad46dda59ae9d14b11ac1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:04.754880 15995 tablet_replica.cc:330] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:04.755059 15995 raft_consensus.cc:2229] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.755249 12710 tablet_service.cc:1514] Processing DeleteTablet for tablet acf6ca36b79e4c69a61892d51978baa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:04.755280 15995 raft_consensus.cc:2258] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.755591 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d0ce51e8ad46dda59ae9d14b11ac1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:04.755879 15995 ts_tablet_manager.cc:1875] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.758579 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet acf6ca36b79e4c69a61892d51978baa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:04.758627 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d0ce51e8ad46dda59ae9d14b11ac1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:04 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:04.759120 15993 ts_tablet_manager.cc:1888] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.759676 15993 log.cc:1196] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/17bd47b83e7f42698a2071cf82c6aa94
I20240502 08:18:04.760633 15993 ts_tablet_manager.cc:1909] T 17bd47b83e7f42698a2071cf82c6aa94 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:04.761584 15956 raft_consensus.cc:686] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:04.761665 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 17bd47b83e7f42698a2071cf82c6aa94 (table rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]) successfully deleted
I20240502 08:18:04.761907 15995 ts_tablet_manager.cc:1888] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.761948 15996 ts_tablet_manager.cc:1888] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.762050 15995 log.cc:1196] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/97d0ce51e8ad46dda59ae9d14b11ac1a
I20240502 08:18:04.762151 15996 log.cc:1196] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/acf6ca36b79e4c69a61892d51978baa5
I20240502 08:18:04.762284 15956 consensus_queue.cc:237] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:04.762631 15996 ts_tablet_manager.cc:1909] T acf6ca36b79e4c69a61892d51978baa5 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:04.763757 15994 ts_tablet_manager.cc:1888] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.763938 15995 ts_tablet_manager.cc:1909] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:04.763960 15994 log.cc:1196] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/17bd47b83e7f42698a2071cf82c6aa94
I20240502 08:18:04.764109 15993 tablet_replica.cc:330] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:04.764329 15993 raft_consensus.cc:2229] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.764367 15994 ts_tablet_manager.cc:1909] T 17bd47b83e7f42698a2071cf82c6aa94 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:04.764513 15993 raft_consensus.cc:2258] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.764771 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:04.765211 15993 ts_tablet_manager.cc:1875] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.765699 15994 tablet_replica.cc:330] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:04.765923 15994 raft_consensus.cc:2229] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.766196 15994 raft_consensus.cc:2258] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.766538 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 97d0ce51e8ad46dda59ae9d14b11ac1a (table rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]) successfully deleted
I20240502 08:18:04.766782 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 17bd47b83e7f42698a2071cf82c6aa94 (table rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]) successfully deleted
I20240502 08:18:04.767016 15994 ts_tablet_manager.cc:1875] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.769191 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet acf6ca36b79e4c69a61892d51978baa5 (table rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]) successfully deleted
I20240502 08:18:04.770066 15993 ts_tablet_manager.cc:1888] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.770184 15993 log.cc:1196] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/97d0ce51e8ad46dda59ae9d14b11ac1a
I20240502 08:18:04.770983 15488 raft_consensus.cc:1261] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:04.771725 16008 consensus_queue.cc:1035] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:04.772049 15886 raft_consensus.cc:1261] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:04.772684 16008 consensus_queue.cc:1035] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:04.774809 15993 ts_tablet_manager.cc:1909] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:04.777448 15993 tablet_replica.cc:330] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:04.777593 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 97d0ce51e8ad46dda59ae9d14b11ac1a (table rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]) successfully deleted
I20240502 08:18:04.777698 15993 raft_consensus.cc:2229] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:04.778443 15993 raft_consensus.cc:2258] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.779187 15993 ts_tablet_manager.cc:1875] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.781054 12579 catalog_manager.cc:5484] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:04.785064 15993 ts_tablet_manager.cc:1888] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.785913 15993 log.cc:1196] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/acf6ca36b79e4c69a61892d51978baa5
I20240502 08:18:04.786366 15993 ts_tablet_manager.cc:1909] T acf6ca36b79e4c69a61892d51978baa5 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:04.787015 15994 ts_tablet_manager.cc:1888] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.787644 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet acf6ca36b79e4c69a61892d51978baa5 (table rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]) successfully deleted
I20240502 08:18:04.790149 15994 log.cc:1196] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/acf6ca36b79e4c69a61892d51978baa5
I20240502 08:18:04.791636 15994 ts_tablet_manager.cc:1909] T acf6ca36b79e4c69a61892d51978baa5 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:04.792654 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet acf6ca36b79e4c69a61892d51978baa5 (table rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]) successfully deleted
I20240502 08:18:04.792960 12579 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet 93bd16eac6764d0bb5e1389ba990ff4b (table rebalancer_dynamic_table [id=7d8c94f188c34464b68f2904c54b4712]). Expected version 1 got 0
I20240502 08:18:04.793879 15967 tablet.cc:1722] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:04.794286 15967 tablet.cc:1620] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:04.794876 16059 mvcc.cc:205] Tried to move back new op lower bound from 7023156776014094336 to 7023156775437692928. Current Snapshot: MvccSnapshot[applied={T|T < 7023156776014094336}]
I20240502 08:18:04.795509 12579 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 93bd16eac6764d0bb5e1389ba990ff4b (table rebalancer_dynamic_table [id=7d8c94f188c34464b68f2904c54b4712]). Expected version 1 got 0
I20240502 08:18:04.804121 12586 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=7d8c94f188c34464b68f2904c54b4712] alter complete (version 1)
I20240502 08:18:04.810639 16064 tablet.cc:1722] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:04.811323 15994 tablet_replica.cc:330] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:04.811579 15994 raft_consensus.cc:2229] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:04.811952 12579 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 93bd16eac6764d0bb5e1389ba990ff4b (table rebalancer_dynamic_table [id=7d8c94f188c34464b68f2904c54b4712]). Expected version 1 got 0
I20240502 08:18:04.812366 15994 raft_consensus.cc:2258] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:04.813089 15994 ts_tablet_manager.cc:1875] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:04.814144 16063 tablet.cc:1722] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:04.815748 12579 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet 93bd16eac6764d0bb5e1389ba990ff4b (table rebalancer_dynamic_table [id=7d8c94f188c34464b68f2904c54b4712]). Expected version 1 got 0
I20240502 08:18:04.817101 16064 tablet.cc:1620] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:04.817456 16063 tablet.cc:1620] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:04.836580 15994 ts_tablet_manager.cc:1888] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:04.839110 15994 log.cc:1196] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/97d0ce51e8ad46dda59ae9d14b11ac1a
I20240502 08:18:04.840353 15994 ts_tablet_manager.cc:1909] T 97d0ce51e8ad46dda59ae9d14b11ac1a P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:04.842574 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 97d0ce51e8ad46dda59ae9d14b11ac1a (table rebalance_test_table_2 [id=24185a89a71d4c748c79f3a21aea8a0b]) successfully deleted
I20240502 08:18:04.859663 15867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:04.874274 12709 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:04.895272 15601 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:18:04.909822 12586 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:04.919129 13101 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 08:18:04.919610 15734 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:04.922075 15967 tablet.cc:1722] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:04.923179 15967 tablet.cc:1620] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:04.927042 16063 tablet.cc:1722] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:04.927569 16063 tablet.cc:1620] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:04.931648 16064 tablet.cc:1722] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:04.932116 16064 tablet.cc:1620] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:04.945698 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=7d8c94f188c34464b68f2904c54b4712] alter complete (version 2)
I20240502 08:18:04.948012 16008 consensus_queue.cc:1035] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:04.960774 15955 consensus_queue.cc:1035] T 9e51af7112424e52a9e9c3200a30f86d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:04.965802 12972 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:18:04.993860 15468 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:04.994246 12841 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:05.048158 12586 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:05.059859 15967 tablet.cc:1722] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:05.066361 16063 tablet.cc:1722] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:05.066675 16064 tablet.cc:1722] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:05.074986 12586 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=7d8c94f188c34464b68f2904c54b4712] alter complete (version 3)
I20240502 08:18:05.124181 16010 consensus_queue.cc:1035] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.129587 15422 consensus_queue.cc:1035] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.179147 12583 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:05.179350 12583 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:05.184092 12583 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 93bd16eac6764d0bb5e1389ba990ff4b
I20240502 08:18:05.185117 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet 93bd16eac6764d0bb5e1389ba990ff4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:05.185459 15601 tablet_service.cc:1514] Processing DeleteTablet for tablet 93bd16eac6764d0bb5e1389ba990ff4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:05.185464 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet 93bd16eac6764d0bb5e1389ba990ff4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:05.190989 15956 consensus_queue.cc:1035] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.196514 16121 tablet_replica.cc:330] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:05.196704 16120 tablet_replica.cc:330] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:05.197127 16122 tablet_replica.cc:330] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:05.197206 16120 raft_consensus.cc:2229] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.198172 16122 raft_consensus.cc:2229] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.198446 16121 raft_consensus.cc:2229] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:05.198896 16121 raft_consensus.cc:2258] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.199340 16122 raft_consensus.cc:2258] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.201305 16121 ts_tablet_manager.cc:1875] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.201977 16120 raft_consensus.cc:2258] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.204022 16122 ts_tablet_manager.cc:1875] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.211350 16120 ts_tablet_manager.cc:1875] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.217936 15956 consensus_queue.cc:1035] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.223526 16120 ts_tablet_manager.cc:1888] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:05.223696 16120 log.cc:1196] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/93bd16eac6764d0bb5e1389ba990ff4b
I20240502 08:18:05.224231 16120 ts_tablet_manager.cc:1909] T 93bd16eac6764d0bb5e1389ba990ff4b P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:05.225497 16121 ts_tablet_manager.cc:1888] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:05.225674 16121 log.cc:1196] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/93bd16eac6764d0bb5e1389ba990ff4b
I20240502 08:18:05.226032 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 93bd16eac6764d0bb5e1389ba990ff4b (table rebalancer_dynamic_table_new_name [id=7d8c94f188c34464b68f2904c54b4712]) successfully deleted
I20240502 08:18:05.226573 16121 ts_tablet_manager.cc:1909] T 93bd16eac6764d0bb5e1389ba990ff4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting consensus metadata
I20240502 08:18:05.227080 16122 ts_tablet_manager.cc:1888] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:05.228029 16122 log.cc:1196] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/93bd16eac6764d0bb5e1389ba990ff4b
I20240502 08:18:05.228345 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 93bd16eac6764d0bb5e1389ba990ff4b (table rebalancer_dynamic_table_new_name [id=7d8c94f188c34464b68f2904c54b4712]) successfully deleted
I20240502 08:18:05.229009 16122 ts_tablet_manager.cc:1909] T 93bd16eac6764d0bb5e1389ba990ff4b P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:05.230947 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 93bd16eac6764d0bb5e1389ba990ff4b (table rebalancer_dynamic_table_new_name [id=7d8c94f188c34464b68f2904c54b4712]) successfully deleted
I20240502 08:18:05.253451 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 08:18:05.253629 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 08:18:05.257000 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet afea0bbd4c0d41d4a63d33b21f542038
I20240502 08:18:05.257632 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet c3d9fdffe8fb40009ca236f998ee9dba
I20240502 08:18:05.257663 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet afea0bbd4c0d41d4a63d33b21f542038 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:05.257871 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet afea0bbd4c0d41d4a63d33b21f542038 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:05.257946 15994 tablet_replica.cc:330] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:05.257972 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet afea0bbd4c0d41d4a63d33b21f542038 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:05.258108 15993 tablet_replica.cc:330] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:05.258155 15994 raft_consensus.cc:2229] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.258229 15996 tablet_replica.cc:330] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:05.258297 15993 raft_consensus.cc:2229] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:05.258383 15994 raft_consensus.cc:2258] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.258414 15996 raft_consensus.cc:2229] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.258622 15993 raft_consensus.cc:2258] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.258666 15996 raft_consensus.cc:2258] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.259135 15994 ts_tablet_manager.cc:1875] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.259267 15996 ts_tablet_manager.cc:1875] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.259299 15993 ts_tablet_manager.cc:1875] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.265498 15993 ts_tablet_manager.cc:1888] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.265641 15993 log.cc:1196] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/afea0bbd4c0d41d4a63d33b21f542038
I20240502 08:18:05.265813 15994 ts_tablet_manager.cc:1888] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.265851 15996 ts_tablet_manager.cc:1888] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.265959 15996 log.cc:1196] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/afea0bbd4c0d41d4a63d33b21f542038
I20240502 08:18:05.266018 15994 log.cc:1196] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/afea0bbd4c0d41d4a63d33b21f542038
I20240502 08:18:05.266027 15993 ts_tablet_manager.cc:1909] T afea0bbd4c0d41d4a63d33b21f542038 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:05.266402 15996 ts_tablet_manager.cc:1909] T afea0bbd4c0d41d4a63d33b21f542038 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:05.266424 15994 ts_tablet_manager.cc:1909] T afea0bbd4c0d41d4a63d33b21f542038 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:05.267019 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet afea0bbd4c0d41d4a63d33b21f542038 (table rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]) successfully deleted
I20240502 08:18:05.267196 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet c52f670b014a4c608807fa9b8877d481
I20240502 08:18:05.267621 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet afea0bbd4c0d41d4a63d33b21f542038 (table rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]) successfully deleted
I20240502 08:18:05.267990 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d9fdffe8fb40009ca236f998ee9dba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:05.268213 15996 tablet_replica.cc:330] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:05.268436 15996 raft_consensus.cc:2229] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.268657 15996 raft_consensus.cc:2258] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.268687 12972 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d9fdffe8fb40009ca236f998ee9dba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:05.268724 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet c52f670b014a4c608807fa9b8877d481 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:05.269205 16131 tablet_replica.cc:330] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:05.269340 15996 ts_tablet_manager.cc:1875] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.269461 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet c52f670b014a4c608807fa9b8877d481 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:05.269518 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d9fdffe8fb40009ca236f998ee9dba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:05.269572 12710 tablet_service.cc:1514] Processing DeleteTablet for tablet c52f670b014a4c608807fa9b8877d481 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:05.269676 15993 tablet_replica.cc:330] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:05.270521 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet afea0bbd4c0d41d4a63d33b21f542038 (table rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]) successfully deleted
I20240502 08:18:05.272998 15994 tablet_replica.cc:330] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:05.273181 15994 raft_consensus.cc:2229] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.273398 15994 raft_consensus.cc:2258] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.273993 15994 ts_tablet_manager.cc:1875] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.275447 15996 ts_tablet_manager.cc:1888] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.275625 15996 log.cc:1196] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/c3d9fdffe8fb40009ca236f998ee9dba
I20240502 08:18:05.276036 15996 ts_tablet_manager.cc:1909] T c3d9fdffe8fb40009ca236f998ee9dba P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:05.276193 15993 raft_consensus.cc:2229] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:05.277150 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet c3d9fdffe8fb40009ca236f998ee9dba (table rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]) successfully deleted
I20240502 08:18:05.277534 15993 raft_consensus.cc:2258] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.278183 15993 ts_tablet_manager.cc:1875] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.283311 15994 ts_tablet_manager.cc:1888] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.283633 15994 log.cc:1196] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/c3d9fdffe8fb40009ca236f998ee9dba
I20240502 08:18:05.284296 15994 ts_tablet_manager.cc:1909] T c3d9fdffe8fb40009ca236f998ee9dba P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:05.284987 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet c3d9fdffe8fb40009ca236f998ee9dba (table rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]) successfully deleted
I20240502 08:18:05.285158 15994 tablet_replica.cc:330] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:05.285347 15994 raft_consensus.cc:2229] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.285511 15994 raft_consensus.cc:2258] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.286026 15994 ts_tablet_manager.cc:1875] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.286303 16131 raft_consensus.cc:2229] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:05.286711 16131 raft_consensus.cc:2258] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.287187 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:05.290334 15993 ts_tablet_manager.cc:1888] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.290448 15993 log.cc:1196] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/c52f670b014a4c608807fa9b8877d481
I20240502 08:18:05.290854 15993 ts_tablet_manager.cc:1909] T c52f670b014a4c608807fa9b8877d481 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:05.291514 16131 ts_tablet_manager.cc:1875] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.296855 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 4c71dfcfb4b14e948ee724c305866522 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e47823ae74d4ff7b13bea0531621e7c]), partition=
I20240502 08:18:05.297111 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 4c71dfcfb4b14e948ee724c305866522 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e47823ae74d4ff7b13bea0531621e7c]), partition=
I20240502 08:18:05.297201 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c71dfcfb4b14e948ee724c305866522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.297423 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c71dfcfb4b14e948ee724c305866522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.297576 16131 ts_tablet_manager.cc:1888] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.297677 16131 log.cc:1196] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/c3d9fdffe8fb40009ca236f998ee9dba
I20240502 08:18:05.298076 16131 ts_tablet_manager.cc:1909] T c3d9fdffe8fb40009ca236f998ee9dba P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:05.301012 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 4c71dfcfb4b14e948ee724c305866522 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e47823ae74d4ff7b13bea0531621e7c]), partition=
I20240502 08:18:05.301410 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c71dfcfb4b14e948ee724c305866522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.304316 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet c52f670b014a4c608807fa9b8877d481 (table rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]) successfully deleted
I20240502 08:18:05.306986 16131 tablet_replica.cc:330] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:05.307235 16131 raft_consensus.cc:2229] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.307480 16131 raft_consensus.cc:2258] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.307507 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet c3d9fdffe8fb40009ca236f998ee9dba (table rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]) successfully deleted
I20240502 08:18:05.308207 16131 ts_tablet_manager.cc:1875] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.314388 16132 tablet_bootstrap.cc:492] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:05.317147 16131 ts_tablet_manager.cc:1888] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.317291 16132 tablet_bootstrap.cc:654] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.319015 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:05.322571 16131 log.cc:1196] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/c52f670b014a4c608807fa9b8877d481
I20240502 08:18:05.322993 16131 ts_tablet_manager.cc:1909] T c52f670b014a4c608807fa9b8877d481 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:05.324473 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet c52f670b014a4c608807fa9b8877d481 (table rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]) successfully deleted
I20240502 08:18:05.329913 16132 tablet_bootstrap.cc:492] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:05.330053 16132 ts_tablet_manager.cc:1367] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 08:18:05.330860 16132 raft_consensus.cc:348] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.331338 16132 raft_consensus.cc:374] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.331274 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 939120da41cd41cb83ef225b9942f82e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ab84e3c981dc4fcf95f4296f05da39ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.331449 16132 raft_consensus.cc:729] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.331724 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939120da41cd41cb83ef225b9942f82e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.331758 16132 consensus_queue.cc:260] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.331799 15866 tablet_service.cc:1467] Processing CreateTablet for tablet 939120da41cd41cb83ef225b9942f82e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ab84e3c981dc4fcf95f4296f05da39ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.332240 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939120da41cd41cb83ef225b9942f82e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.334548 16132 ts_tablet_manager.cc:1398] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:05.335592 15602 tablet_service.cc:1467] Processing CreateTablet for tablet 939120da41cd41cb83ef225b9942f82e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ab84e3c981dc4fcf95f4296f05da39ba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.335994 15602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939120da41cd41cb83ef225b9942f82e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.338888 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 040878c4c4ae4684b02296ee9eac8985 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ab84e3c981dc4fcf95f4296f05da39ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.338937 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 040878c4c4ae4684b02296ee9eac8985 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ab84e3c981dc4fcf95f4296f05da39ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.339308 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 040878c4c4ae4684b02296ee9eac8985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.339334 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 040878c4c4ae4684b02296ee9eac8985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.339783 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 65cf482906a646aabd59f22900ece16f (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ab84e3c981dc4fcf95f4296f05da39ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.340145 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cf482906a646aabd59f22900ece16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.343441 15600 tablet_service.cc:1467] Processing CreateTablet for tablet 040878c4c4ae4684b02296ee9eac8985 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ab84e3c981dc4fcf95f4296f05da39ba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.343776 15600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 040878c4c4ae4684b02296ee9eac8985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.344887 16132 tablet_bootstrap.cc:492] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:05.346086 16132 tablet_bootstrap.cc:654] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.346817 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 65cf482906a646aabd59f22900ece16f (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ab84e3c981dc4fcf95f4296f05da39ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.350840 15466 tablet_service.cc:1467] Processing CreateTablet for tablet 65cf482906a646aabd59f22900ece16f (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ab84e3c981dc4fcf95f4296f05da39ba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:05.351300 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cf482906a646aabd59f22900ece16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.357604 16136 tablet_bootstrap.cc:492] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:05.359750 16136 tablet_bootstrap.cc:654] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.362143 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65cf482906a646aabd59f22900ece16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:05.363049 16132 tablet_bootstrap.cc:492] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:05.363329 16132 ts_tablet_manager.cc:1367] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.010s
I20240502 08:18:05.364112 16132 raft_consensus.cc:348] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.364382 16132 raft_consensus.cc:374] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.364473 16132 raft_consensus.cc:729] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.364732 16132 consensus_queue.cc:260] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.368388 16132 ts_tablet_manager.cc:1398] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:18:05.370643 16132 tablet_bootstrap.cc:492] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:05.372095 16132 tablet_bootstrap.cc:654] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.386343 15994 ts_tablet_manager.cc:1888] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.386582 15994 log.cc:1196] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/c52f670b014a4c608807fa9b8877d481
I20240502 08:18:05.387219 15994 ts_tablet_manager.cc:1909] T c52f670b014a4c608807fa9b8877d481 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:05.388398 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet c52f670b014a4c608807fa9b8877d481 (table rebalance_test_table_4 [id=fdb1fb8a12f046ea93840594609f50d5]) successfully deleted
I20240502 08:18:05.389266 16133 tablet_bootstrap.cc:492] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:05.397948 16024 raft_consensus.cc:482] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.398092 16024 raft_consensus.cc:504] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.398844 16134 tablet_bootstrap.cc:492] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:05.400015 16024 leader_election.cc:288] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:05.403807 16136 tablet_bootstrap.cc:492] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:05.403964 16136 ts_tablet_manager.cc:1367] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.013s
I20240502 08:18:05.404896 16136 raft_consensus.cc:348] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:05.405118 16136 raft_consensus.cc:374] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.405203 16136 raft_consensus.cc:729] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.405421 16136 consensus_queue.cc:260] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:05.405941 16136 ts_tablet_manager.cc:1398] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:05.406224 16136 tablet_bootstrap.cc:492] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:05.408046 16136 tablet_bootstrap.cc:654] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.410362 16134 tablet_bootstrap.cc:654] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.412501 16133 tablet_bootstrap.cc:654] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.413204 16136 tablet_bootstrap.cc:492] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:05.413416 16136 ts_tablet_manager.cc:1367] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 08:18:05.414168 16136 raft_consensus.cc:348] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.414451 16136 raft_consensus.cc:374] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.414587 16136 raft_consensus.cc:729] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.414853 16136 consensus_queue.cc:260] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.416556 16134 tablet_bootstrap.cc:492] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:05.416743 16134 ts_tablet_manager.cc:1367] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 08:18:05.417680 16134 raft_consensus.cc:348] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.422190 16134 raft_consensus.cc:374] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.422333 16134 raft_consensus.cc:729] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.422762 16134 consensus_queue.cc:260] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.423540 16025 raft_consensus.cc:482] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.423666 16025 raft_consensus.cc:504] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.424079 16132 tablet_bootstrap.cc:492] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:05.424212 16132 ts_tablet_manager.cc:1367] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.054s user 0.029s sys 0.007s
I20240502 08:18:05.424338 16136 ts_tablet_manager.cc:1398] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240502 08:18:05.424813 16025 leader_election.cc:288] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:05.424926 16132 raft_consensus.cc:348] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.425101 16132 raft_consensus.cc:374] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.425184 16132 raft_consensus.cc:729] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.425412 16132 consensus_queue.cc:260] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.432914 16132 ts_tablet_manager.cc:1398] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240502 08:18:05.436105 16134 ts_tablet_manager.cc:1398] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 08:18:05.436695 16134 tablet_bootstrap.cc:492] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:05.438297 16134 tablet_bootstrap.cc:654] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.445267 16133 tablet_bootstrap.cc:492] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:05.445475 16133 ts_tablet_manager.cc:1367] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.006s
I20240502 08:18:05.446341 16133 raft_consensus.cc:348] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.446658 16133 raft_consensus.cc:374] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.446846 16133 raft_consensus.cc:729] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.447108 16133 consensus_queue.cc:260] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.447671 16133 ts_tablet_manager.cc:1398] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:05.447654 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c71dfcfb4b14e948ee724c305866522" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:05.447890 15886 raft_consensus.cc:2454] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:05.447988 16133 tablet_bootstrap.cc:492] T 939120da41cd41cb83ef225b9942f82e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:05.451825 15420 leader_election.cc:302] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:05.452163 16025 raft_consensus.cc:2781] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:05.452266 16025 raft_consensus.cc:482] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:05.452344 16025 raft_consensus.cc:3037] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.452821 16133 tablet_bootstrap.cc:654] T 939120da41cd41cb83ef225b9942f82e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.455291 16025 raft_consensus.cc:504] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.455628 15956 raft_consensus.cc:482] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.455798 15956 raft_consensus.cc:504] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:05.455917 16025 leader_election.cc:288] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:05.456555 15956 leader_election.cc:288] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:05.457758 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cf482906a646aabd59f22900ece16f" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
W20240502 08:18:05.458397 15690 leader_election.cc:341] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:05.460481 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c71dfcfb4b14e948ee724c305866522" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:05.460675 15886 raft_consensus.cc:3037] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.463261 16134 tablet_bootstrap.cc:492] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:05.463479 16134 ts_tablet_manager.cc:1367] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.007s
I20240502 08:18:05.463510 15886 raft_consensus.cc:2454] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:05.464041 15420 leader_election.cc:302] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:05.464332 16025 raft_consensus.cc:2781] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:05.464361 16134 raft_consensus.cc:348] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:05.466339 16134 raft_consensus.cc:374] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.468047 16134 raft_consensus.cc:729] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.468057 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c71dfcfb4b14e948ee724c305866522" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:05.468281 15622 raft_consensus.cc:2454] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:05.468453 16134 consensus_queue.cc:260] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:05.468923 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c71dfcfb4b14e948ee724c305866522" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:05.469102 15622 raft_consensus.cc:3037] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.472051 15622 raft_consensus.cc:2454] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:05.474725 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cf482906a646aabd59f22900ece16f" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:05.474953 15487 raft_consensus.cc:2454] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:05.475510 15688 leader_election.cc:302] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:05.476075 15956 raft_consensus.cc:2781] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:05.476182 15956 raft_consensus.cc:482] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:05.476295 15956 raft_consensus.cc:3037] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.479118 15956 raft_consensus.cc:504] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:05.480113 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cf482906a646aabd59f22900ece16f" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:05.480285 15956 leader_election.cc:288] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
W20240502 08:18:05.480587 15690 leader_election.cc:341] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:05.481236 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cf482906a646aabd59f22900ece16f" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:05.481441 15487 raft_consensus.cc:3037] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.484750 15487 raft_consensus.cc:2454] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:05.485251 15688 leader_election.cc:302] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:05.485745 15956 raft_consensus.cc:2781] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:05.486289 15956 raft_consensus.cc:686] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:05.486666 15956 consensus_queue.cc:237] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:05.490845 16057 raft_consensus.cc:482] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.490972 16057 raft_consensus.cc:504] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.491709 16057 leader_election.cc:288] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:05.492265 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
W20240502 08:18:05.492902 15822 leader_election.cc:341] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:05.493656 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:05.494027 16134 ts_tablet_manager.cc:1398] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.030s user 0.001s sys 0.003s
W20240502 08:18:05.494151 15690 leader_election.cc:341] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:05.497428 12583 catalog_manager.cc:5484] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:05.501335 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:05.501543 15755 raft_consensus.cc:2454] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:05.501583 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:05.501801 15886 raft_consensus.cc:2454] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:05.502132 15689 leader_election.cc:302] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:05.502555 16057 raft_consensus.cc:2781] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:05.502648 16057 raft_consensus.cc:482] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:05.502724 16057 raft_consensus.cc:3037] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.503445 16134 tablet_bootstrap.cc:492] T 939120da41cd41cb83ef225b9942f82e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:05.505023 16134 tablet_bootstrap.cc:654] T 939120da41cd41cb83ef225b9942f82e P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.505319 16057 raft_consensus.cc:504] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.506381 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:05.506765 15818 leader_election.cc:302] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:05.506896 16133 tablet_bootstrap.cc:492] T 939120da41cd41cb83ef225b9942f82e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:05.507056 16133 ts_tablet_manager.cc:1367] T 939120da41cd41cb83ef225b9942f82e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.059s user 0.013s sys 0.024s
I20240502 08:18:05.507462 16024 raft_consensus.cc:2781] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:05.507562 16024 raft_consensus.cc:482] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:05.507638 16024 raft_consensus.cc:3037] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.507838 16133 raft_consensus.cc:348] T 939120da41cd41cb83ef225b9942f82e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.508049 16133 raft_consensus.cc:374] T 939120da41cd41cb83ef225b9942f82e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.508109 16133 raft_consensus.cc:729] T 939120da41cd41cb83ef225b9942f82e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.508340 16133 consensus_queue.cc:260] T 939120da41cd41cb83ef225b9942f82e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.509642 16133 ts_tablet_manager.cc:1398] T 939120da41cd41cb83ef225b9942f82e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:05.509912 16133 tablet_bootstrap.cc:492] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:05.510104 16024 raft_consensus.cc:504] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.510666 16025 raft_consensus.cc:686] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
W20240502 08:18:05.510850 15690 leader_election.cc:341] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:05.511088 16025 consensus_queue.cc:237] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.512192 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:05.512430 15886 raft_consensus.cc:2379] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9333cb10227b4801b4fa7c24e22e131b in current term 1: Already voted for candidate 10f0e36203914b85b77e78609ffced98 in this term.
I20240502 08:18:05.513355 16133 tablet_bootstrap.cc:654] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.513491 15689 leader_election.cc:302] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b; no voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:05.513823 15956 raft_consensus.cc:2726] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:18:05.514613 16057 leader_election.cc:288] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:05.515506 16024 leader_election.cc:288] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:05.515910 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:05.516075 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:05.516333 15755 raft_consensus.cc:2379] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10f0e36203914b85b77e78609ffced98 in current term 1: Already voted for candidate 9333cb10227b4801b4fa7c24e22e131b in this term.
W20240502 08:18:05.516355 15822 leader_election.cc:341] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:05.517131 15818 leader_election.cc:302] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10f0e36203914b85b77e78609ffced98; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:05.517769 16024 raft_consensus.cc:2726] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:18:05.519966 12583 catalog_manager.cc:5484] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:05.521378 16134 tablet_bootstrap.cc:492] T 939120da41cd41cb83ef225b9942f82e P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:05.522501 16134 ts_tablet_manager.cc:1367] T 939120da41cd41cb83ef225b9942f82e P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
I20240502 08:18:05.523275 16134 raft_consensus.cc:348] T 939120da41cd41cb83ef225b9942f82e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.523489 16134 raft_consensus.cc:374] T 939120da41cd41cb83ef225b9942f82e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.523599 16134 raft_consensus.cc:729] T 939120da41cd41cb83ef225b9942f82e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.523872 16133 tablet_bootstrap.cc:492] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:05.523844 16134 consensus_queue.cc:260] T 939120da41cd41cb83ef225b9942f82e P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.523998 16133 ts_tablet_manager.cc:1367] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.002s
I20240502 08:18:05.524346 16134 ts_tablet_manager.cc:1398] T 939120da41cd41cb83ef225b9942f82e P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:05.524696 16133 raft_consensus.cc:348] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:05.524868 16133 raft_consensus.cc:374] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.524935 16133 raft_consensus.cc:729] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.525095 16133 consensus_queue.cc:260] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:05.525609 16133 ts_tablet_manager.cc:1398] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 08:18:05.525884 16133 tablet_bootstrap.cc:492] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:05.527135 16133 tablet_bootstrap.cc:654] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:05.528849 16133 tablet_bootstrap.cc:492] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:05.529048 16133 ts_tablet_manager.cc:1367] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 08:18:05.529546 16133 raft_consensus.cc:348] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.529690 16133 raft_consensus.cc:374] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:05.529742 16133 raft_consensus.cc:729] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:05.529918 16133 consensus_queue.cc:260] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.530357 16133 ts_tablet_manager.cc:1398] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:18:05.573630 15955 raft_consensus.cc:482] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.573792 15955 raft_consensus.cc:504] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:05.574308 15955 leader_election.cc:288] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:05.574754 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cf482906a646aabd59f22900ece16f" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:05.574847 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65cf482906a646aabd59f22900ece16f" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:05.575073 15487 raft_consensus.cc:2379] T 65cf482906a646aabd59f22900ece16f P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in current term 1: Already voted for candidate 9333cb10227b4801b4fa7c24e22e131b in this term.
I20240502 08:18:05.575410 15554 leader_election.cc:302] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:05.575673 15955 raft_consensus.cc:3037] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.577611 15955 raft_consensus.cc:2726] T 65cf482906a646aabd59f22900ece16f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:05.610141 16024 raft_consensus.cc:482] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.610301 16024 raft_consensus.cc:504] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.611060 16024 leader_election.cc:288] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:05.611685 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939120da41cd41cb83ef225b9942f82e" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:05.611912 15622 raft_consensus.cc:2454] T 939120da41cd41cb83ef225b9942f82e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:05.612289 15822 leader_election.cc:302] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:05.612592 16024 raft_consensus.cc:2781] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:05.612690 16024 raft_consensus.cc:482] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:05.612771 16024 raft_consensus.cc:3037] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.615531 16024 raft_consensus.cc:504] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.616549 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939120da41cd41cb83ef225b9942f82e" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:05.616712 15622 raft_consensus.cc:3037] T 939120da41cd41cb83ef225b9942f82e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.616741 16024 leader_election.cc:288] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:05.617270 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939120da41cd41cb83ef225b9942f82e" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:05.617391 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939120da41cd41cb83ef225b9942f82e" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:05.617516 15487 raft_consensus.cc:2454] T 939120da41cd41cb83ef225b9942f82e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:05.619284 15622 raft_consensus.cc:2454] T 939120da41cd41cb83ef225b9942f82e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:05.619684 15822 leader_election.cc:302] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:05.619974 16024 raft_consensus.cc:2781] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:05.620332 16024 raft_consensus.cc:686] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:05.620707 16024 consensus_queue.cc:237] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.623448 12584 catalog_manager.cc:5484] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:05.652639 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:05.658557 15622 raft_consensus.cc:1261] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:05.658797 15886 raft_consensus.cc:1261] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:05.659113 16025 consensus_queue.cc:1035] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.659396 16152 consensus_queue.cc:1035] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.662942 12584 catalog_manager.cc:5576] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983) does not have the latest schema for tablet 4c71dfcfb4b14e948ee724c305866522 (table rebalancer_dynamic_table [id=2e47823ae74d4ff7b13bea0531621e7c]). Expected version 1 got 0
I20240502 08:18:05.666373 12583 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 4c71dfcfb4b14e948ee724c305866522 (table rebalancer_dynamic_table [id=2e47823ae74d4ff7b13bea0531621e7c]). Expected version 1 got 0
I20240502 08:18:05.666746 16165 tablet.cc:1722] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:05.666989 16162 mvcc.cc:205] Tried to move back new op lower bound from 7023156779651174400 to 7023156779057786880. Current Snapshot: MvccSnapshot[applied={T|T < 7023156779651174400}]
I20240502 08:18:05.667872 16165 tablet.cc:1620] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:05.668821 16167 tablet.cc:1722] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:05.670101 12583 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 4c71dfcfb4b14e948ee724c305866522 (table rebalancer_dynamic_table [id=2e47823ae74d4ff7b13bea0531621e7c]). Expected version 1 got 0
I20240502 08:18:05.670047 16166 tablet.cc:1722] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:05.672317 12583 catalog_manager.cc:5576] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983) does not have the latest schema for tablet 4c71dfcfb4b14e948ee724c305866522 (table rebalancer_dynamic_table [id=2e47823ae74d4ff7b13bea0531621e7c]). Expected version 1 got 0
I20240502 08:18:05.673151 16167 tablet.cc:1620] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:05.676867 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=2e47823ae74d4ff7b13bea0531621e7c] alter complete (version 1)
I20240502 08:18:05.676921 16166 tablet.cc:1620] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:05.769071 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 08:18:05.769313 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 08:18:05.772426 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 6ef7e83665894c1cb10cd470229cc417
I20240502 08:18:05.773094 12972 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef7e83665894c1cb10cd470229cc417 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:05.772922 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce
I20240502 08:18:05.773269 12710 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef7e83665894c1cb10cd470229cc417 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:05.773330 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef7e83665894c1cb10cd470229cc417 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:05.773439 16131 tablet_replica.cc:330] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:05.773607 15994 tablet_replica.cc:330] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:05.773630 16131 raft_consensus.cc:2229] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.773736 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet c662cee7c7bc4465b3c3050a3e64eea6
I20240502 08:18:05.773821 15994 raft_consensus.cc:2229] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.773859 16131 raft_consensus.cc:2258] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.774070 15994 raft_consensus.cc:2258] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.774566 16131 ts_tablet_manager.cc:1875] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.774749 15994 ts_tablet_manager.cc:1875] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.775090 12710 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:05.775182 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet c662cee7c7bc4465b3c3050a3e64eea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:05.775413 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:05.775647 15993 tablet_replica.cc:330] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:05.775664 15996 tablet_replica.cc:330] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:05.775835 15996 raft_consensus.cc:2229] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:05.775846 15993 raft_consensus.cc:2229] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.776053 15993 raft_consensus.cc:2258] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.776152 15996 raft_consensus.cc:2258] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.776721 15993 ts_tablet_manager.cc:1875] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.776759 15996 ts_tablet_manager.cc:1875] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.778640 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet c662cee7c7bc4465b3c3050a3e64eea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:05.779523 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet c662cee7c7bc4465b3c3050a3e64eea6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:05.780934 16131 ts_tablet_manager.cc:1888] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.781086 16131 log.cc:1196] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/6ef7e83665894c1cb10cd470229cc417
I20240502 08:18:05.781529 16131 ts_tablet_manager.cc:1909] T 6ef7e83665894c1cb10cd470229cc417 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:05.781893 12972 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:05 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:05.782191 15994 ts_tablet_manager.cc:1888] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.782485 16131 tablet_replica.cc:330] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:05.782752 16131 raft_consensus.cc:2229] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:05.782373 15994 log.cc:1196] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/6ef7e83665894c1cb10cd470229cc417
I20240502 08:18:05.783170 16131 raft_consensus.cc:2258] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.783327 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 6ef7e83665894c1cb10cd470229cc417 (table rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]) successfully deleted
I20240502 08:18:05.783825 15994 ts_tablet_manager.cc:1909] T 6ef7e83665894c1cb10cd470229cc417 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:05.783735 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:05.783936 16131 ts_tablet_manager.cc:1875] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.784675 15993 ts_tablet_manager.cc:1888] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.784850 15993 log.cc:1196] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/4a1eb8c0e302416bbae1e7cd1e4a86ce
I20240502 08:18:05.785121 15994 tablet_replica.cc:330] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:05.785360 15994 raft_consensus.cc:2229] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:05.786080 15993 ts_tablet_manager.cc:1909] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:05.786319 15994 raft_consensus.cc:2258] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.786705 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 6ef7e83665894c1cb10cd470229cc417 (table rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]) successfully deleted
I20240502 08:18:05.787070 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce (table rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]) successfully deleted
I20240502 08:18:05.790824 16131 ts_tablet_manager.cc:1888] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.790972 15994 ts_tablet_manager.cc:1875] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.790992 16131 log.cc:1196] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/c662cee7c7bc4465b3c3050a3e64eea6
I20240502 08:18:05.791373 16131 ts_tablet_manager.cc:1909] T c662cee7c7bc4465b3c3050a3e64eea6 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:05.792200 16131 tablet_replica.cc:330] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:05.792411 16131 raft_consensus.cc:2229] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.792335 16165 tablet.cc:1722] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:05.792658 16131 raft_consensus.cc:2258] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.793205 16165 tablet.cc:1620] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:05.793306 16131 ts_tablet_manager.cc:1875] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.793380 16167 tablet.cc:1722] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:05.793802 16167 tablet.cc:1620] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:05.794992 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet c662cee7c7bc4465b3c3050a3e64eea6 (table rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]) successfully deleted
I20240502 08:18:05.795581 16166 tablet.cc:1722] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:05.796036 16166 tablet.cc:1620] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:05.803650 16131 ts_tablet_manager.cc:1888] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.803800 16131 log.cc:1196] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/4a1eb8c0e302416bbae1e7cd1e4a86ce
I20240502 08:18:05.804217 16131 ts_tablet_manager.cc:1909] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:05.804597 15996 ts_tablet_manager.cc:1888] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.804792 15996 log.cc:1196] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/6ef7e83665894c1cb10cd470229cc417
I20240502 08:18:05.806154 15996 ts_tablet_manager.cc:1909] T 6ef7e83665894c1cb10cd470229cc417 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:05.806670 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce (table rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]) successfully deleted
I20240502 08:18:05.807348 15996 tablet_replica.cc:330] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:05.807418 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 6ef7e83665894c1cb10cd470229cc417 (table rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]) successfully deleted
I20240502 08:18:05.807574 15996 raft_consensus.cc:2229] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.808370 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=2e47823ae74d4ff7b13bea0531621e7c] alter complete (version 2)
I20240502 08:18:05.808460 15994 ts_tablet_manager.cc:1888] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.808625 15994 log.cc:1196] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/4a1eb8c0e302416bbae1e7cd1e4a86ce
I20240502 08:18:05.808837 15996 raft_consensus.cc:2258] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.809017 15994 ts_tablet_manager.cc:1909] T 4a1eb8c0e302416bbae1e7cd1e4a86ce P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:05.809947 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 4a1eb8c0e302416bbae1e7cd1e4a86ce (table rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]) successfully deleted
I20240502 08:18:05.810125 15994 tablet_replica.cc:330] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:05.810287 15996 ts_tablet_manager.cc:1875] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.810333 15994 raft_consensus.cc:2229] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:05.810591 15994 raft_consensus.cc:2258] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:05.811230 15994 ts_tablet_manager.cc:1875] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:05.815526 15996 ts_tablet_manager.cc:1888] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.815657 15996 log.cc:1196] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/c662cee7c7bc4465b3c3050a3e64eea6
I20240502 08:18:05.815979 15996 ts_tablet_manager.cc:1909] T c662cee7c7bc4465b3c3050a3e64eea6 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:05.816746 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet c662cee7c7bc4465b3c3050a3e64eea6 (table rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]) successfully deleted
I20240502 08:18:05.817111 15994 ts_tablet_manager.cc:1888] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:05.817234 15994 log.cc:1196] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/c662cee7c7bc4465b3c3050a3e64eea6
I20240502 08:18:05.817632 15994 ts_tablet_manager.cc:1909] T c662cee7c7bc4465b3c3050a3e64eea6 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:05.818478 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet c662cee7c7bc4465b3c3050a3e64eea6 (table rebalance_test_table_6 [id=ba97eaf5eb64434d98bb7905da2109a2]) successfully deleted
I20240502 08:18:05.865458 15955 raft_consensus.cc:482] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:05.865649 15955 raft_consensus.cc:504] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:05.866350 15955 leader_election.cc:288] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:05.866734 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:05.866993 15755 raft_consensus.cc:2379] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in current term 1: Already voted for candidate 9333cb10227b4801b4fa7c24e22e131b in this term.
I20240502 08:18:05.866976 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:05.867213 15887 raft_consensus.cc:2379] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in current term 1: Already voted for candidate 10f0e36203914b85b77e78609ffced98 in this term.
I20240502 08:18:05.867622 15555 leader_election.cc:302] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:05.867856 15955 raft_consensus.cc:3037] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:05.869693 15955 raft_consensus.cc:2726] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:05.895884 16057 consensus_queue.cc:1035] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.900028 16057 consensus_queue.cc:1035] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:05.911245 12586 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:05.917855 16165 tablet.cc:1722] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:05.919230 16166 tablet.cc:1722] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:05.919379 16167 tablet.cc:1722] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:05.925985 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=2e47823ae74d4ff7b13bea0531621e7c] alter complete (version 3)
I20240502 08:18:06.003489 15488 raft_consensus.cc:3037] T 939120da41cd41cb83ef225b9942f82e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:06.006072 16024 consensus_queue.cc:1035] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:06.013249 16024 consensus_queue.cc:1035] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:06.027856 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:06.028061 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:06.031211 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 4c71dfcfb4b14e948ee724c305866522
I20240502 08:18:06.032011 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c71dfcfb4b14e948ee724c305866522 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:06.032295 15601 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c71dfcfb4b14e948ee724c305866522 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:06.032364 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c71dfcfb4b14e948ee724c305866522 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:06.032801 16174 tablet_replica.cc:330] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:06.032944 16175 tablet_replica.cc:330] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:06.033051 16174 raft_consensus.cc:2229] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.033151 16175 raft_consensus.cc:2229] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.033357 16174 raft_consensus.cc:2258] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.033387 16175 raft_consensus.cc:2258] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.034233 16174 ts_tablet_manager.cc:1875] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.034233 16175 ts_tablet_manager.cc:1875] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.040414 16175 ts_tablet_manager.cc:1888] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:06.040491 16174 ts_tablet_manager.cc:1888] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:06.040571 16175 log.cc:1196] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/4c71dfcfb4b14e948ee724c305866522
I20240502 08:18:06.040663 16174 log.cc:1196] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/4c71dfcfb4b14e948ee724c305866522
I20240502 08:18:06.040729 16173 tablet_replica.cc:330] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:06.040910 16173 raft_consensus.cc:2229] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:06.040963 16175 ts_tablet_manager.cc:1909] T 4c71dfcfb4b14e948ee724c305866522 P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:06.041067 16174 ts_tablet_manager.cc:1909] T 4c71dfcfb4b14e948ee724c305866522 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting consensus metadata
I20240502 08:18:06.041366 16173 raft_consensus.cc:2258] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.041962 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 4c71dfcfb4b14e948ee724c305866522 (table rebalancer_dynamic_table_new_name [id=2e47823ae74d4ff7b13bea0531621e7c]) successfully deleted
I20240502 08:18:06.042219 16173 ts_tablet_manager.cc:1875] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.042207 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 4c71dfcfb4b14e948ee724c305866522 (table rebalancer_dynamic_table_new_name [id=2e47823ae74d4ff7b13bea0531621e7c]) successfully deleted
I20240502 08:18:06.046814 16173 ts_tablet_manager.cc:1888] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:06.046934 16173 log.cc:1196] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/4c71dfcfb4b14e948ee724c305866522
I20240502 08:18:06.047292 16173 ts_tablet_manager.cc:1909] T 4c71dfcfb4b14e948ee724c305866522 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:06.048058 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 4c71dfcfb4b14e948ee724c305866522 (table rebalancer_dynamic_table_new_name [id=2e47823ae74d4ff7b13bea0531621e7c]) successfully deleted
I20240502 08:18:06.133848 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:06.140367 15734 tablet_service.cc:1467] Processing CreateTablet for tablet dbe55ec22eb14db4808a0fe2054aab93 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=56f4ff84513146ccb41ef079a6a6f11e]), partition=
I20240502 08:18:06.140491 12709 tablet_service.cc:1467] Processing CreateTablet for tablet dbe55ec22eb14db4808a0fe2054aab93 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=56f4ff84513146ccb41ef079a6a6f11e]), partition=
I20240502 08:18:06.140761 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe55ec22eb14db4808a0fe2054aab93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.140815 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe55ec22eb14db4808a0fe2054aab93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.141502 15467 tablet_service.cc:1467] Processing CreateTablet for tablet dbe55ec22eb14db4808a0fe2054aab93 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=56f4ff84513146ccb41ef079a6a6f11e]), partition=
I20240502 08:18:06.141885 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe55ec22eb14db4808a0fe2054aab93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.146652 16176 tablet_bootstrap.cc:492] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:06.148111 16176 tablet_bootstrap.cc:654] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.149644 16176 tablet_bootstrap.cc:492] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:06.149828 16176 ts_tablet_manager.cc:1367] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:18:06.150580 16176 raft_consensus.cc:348] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:06.150830 16176 raft_consensus.cc:374] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.151043 16177 tablet_bootstrap.cc:492] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:06.151098 16176 raft_consensus.cc:729] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.151482 16176 consensus_queue.cc:260] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:06.152316 16176 ts_tablet_manager.cc:1398] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:06.152534 16177 tablet_bootstrap.cc:654] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.154253 16177 tablet_bootstrap.cc:492] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:06.154395 16177 ts_tablet_manager.cc:1367] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:06.155392 16177 raft_consensus.cc:348] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:06.155565 16177 raft_consensus.cc:374] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.155658 16177 raft_consensus.cc:729] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.155944 16177 consensus_queue.cc:260] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:06.156482 16177 ts_tablet_manager.cc:1398] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:06.159752 16178 tablet_bootstrap.cc:492] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:06.161093 16178 tablet_bootstrap.cc:654] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.163019 16178 tablet_bootstrap.cc:492] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:06.163143 16178 ts_tablet_manager.cc:1367] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:06.163727 16178 raft_consensus.cc:348] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:06.163887 16178 raft_consensus.cc:374] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.163972 16178 raft_consensus.cc:729] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.164157 16178 consensus_queue.cc:260] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:06.164640 16178 ts_tablet_manager.cc:1398] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:18:06.275522 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 08:18:06.275769 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 08:18:06.279999 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet bf1c03ab6bf840bf8fc70311bad73440
I20240502 08:18:06.280809 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1c03ab6bf840bf8fc70311bad73440 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:06.281111 15996 tablet_replica.cc:330] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:06.281270 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1c03ab6bf840bf8fc70311bad73440 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:06.281307 15996 raft_consensus.cc:2229] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.281554 15994 tablet_replica.cc:330] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:06.281486 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 5bd28596a64c49139e2b27e2dc68ad68
I20240502 08:18:06.281577 15996 raft_consensus.cc:2258] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.281780 15994 raft_consensus.cc:2229] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.282083 15994 raft_consensus.cc:2258] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.282138 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd28596a64c49139e2b27e2dc68ad68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:06.282313 15996 ts_tablet_manager.cc:1875] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.282490 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet ecfefc1736ae4d5caade9e00af8a3718
I20240502 08:18:06.282750 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd28596a64c49139e2b27e2dc68ad68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:06.282887 15994 ts_tablet_manager.cc:1875] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.283505 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet ecfefc1736ae4d5caade9e00af8a3718 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:06.284198 12972 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd28596a64c49139e2b27e2dc68ad68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:06.284301 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet ecfefc1736ae4d5caade9e00af8a3718 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:06.284536 16131 tablet_replica.cc:330] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:06.284637 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1c03ab6bf840bf8fc70311bad73440 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:06.284632 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet ecfefc1736ae4d5caade9e00af8a3718 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:06.284756 16131 raft_consensus.cc:2229] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.284888 15993 tablet_replica.cc:330] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:06.284999 16131 raft_consensus.cc:2258] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.285059 15993 raft_consensus.cc:2229] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:06.285354 15993 raft_consensus.cc:2258] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.285717 16131 ts_tablet_manager.cc:1875] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.285993 15993 ts_tablet_manager.cc:1875] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.288916 15994 ts_tablet_manager.cc:1888] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.289063 15994 log.cc:1196] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/bf1c03ab6bf840bf8fc70311bad73440
I20240502 08:18:06.289541 15994 ts_tablet_manager.cc:1909] T bf1c03ab6bf840bf8fc70311bad73440 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:06.289716 15996 ts_tablet_manager.cc:1888] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.290019 15996 log.cc:1196] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/bf1c03ab6bf840bf8fc70311bad73440
I20240502 08:18:06.290441 15996 ts_tablet_manager.cc:1909] T bf1c03ab6bf840bf8fc70311bad73440 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:06.290527 15994 tablet_replica.cc:330] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:06.290747 15994 raft_consensus.cc:2229] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.290998 15994 raft_consensus.cc:2258] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.291481 15996 tablet_replica.cc:330] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:06.291617 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet bf1c03ab6bf840bf8fc70311bad73440 (table rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]) successfully deleted
I20240502 08:18:06.291671 15994 ts_tablet_manager.cc:1875] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.291711 15996 raft_consensus.cc:2229] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:06.291895 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet bf1c03ab6bf840bf8fc70311bad73440 (table rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]) successfully deleted
I20240502 08:18:06.292176 16054 raft_consensus.cc:482] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:06.292207 15996 raft_consensus.cc:2258] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.292328 16054 raft_consensus.cc:504] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:06.292675 16131 ts_tablet_manager.cc:1888] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.292932 15996 ts_tablet_manager.cc:1875] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.293320 16131 log.cc:1196] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/5bd28596a64c49139e2b27e2dc68ad68
I20240502 08:18:06.293752 16131 ts_tablet_manager.cc:1909] T 5bd28596a64c49139e2b27e2dc68ad68 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:06.294765 16131 tablet_replica.cc:330] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:06.294867 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 5bd28596a64c49139e2b27e2dc68ad68 (table rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]) successfully deleted
I20240502 08:18:06.295377 16131 raft_consensus.cc:2229] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:06.295735 16131 raft_consensus.cc:2258] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.296425 16131 ts_tablet_manager.cc:1875] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.303102 15996 ts_tablet_manager.cc:1888] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.303275 15996 log.cc:1196] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/5bd28596a64c49139e2b27e2dc68ad68
I20240502 08:18:06.303686 15996 ts_tablet_manager.cc:1909] T 5bd28596a64c49139e2b27e2dc68ad68 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:06.304586 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 5bd28596a64c49139e2b27e2dc68ad68 (table rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]) successfully deleted
I20240502 08:18:06.304839 15996 tablet_replica.cc:330] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:06.305052 15996 raft_consensus.cc:2229] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.305325 15996 raft_consensus.cc:2258] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.305979 15996 ts_tablet_manager.cc:1875] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.307754 16054 leader_election.cc:288] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:06.309386 15994 ts_tablet_manager.cc:1888] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.311475 15994 log.cc:1196] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/5bd28596a64c49139e2b27e2dc68ad68
I20240502 08:18:06.312949 15994 ts_tablet_manager.cc:1909] T 5bd28596a64c49139e2b27e2dc68ad68 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:06.315064 15996 ts_tablet_manager.cc:1888] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.315229 15996 log.cc:1196] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/ecfefc1736ae4d5caade9e00af8a3718
I20240502 08:18:06.315678 15996 ts_tablet_manager.cc:1909] T ecfefc1736ae4d5caade9e00af8a3718 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:06.316586 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet ecfefc1736ae4d5caade9e00af8a3718 (table rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]) successfully deleted
I20240502 08:18:06.319172 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 5bd28596a64c49139e2b27e2dc68ad68 (table rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]) successfully deleted
I20240502 08:18:06.320420 15993 ts_tablet_manager.cc:1888] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.320541 15993 log.cc:1196] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/bf1c03ab6bf840bf8fc70311bad73440
I20240502 08:18:06.321300 16131 ts_tablet_manager.cc:1888] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.321319 15993 ts_tablet_manager.cc:1909] T bf1c03ab6bf840bf8fc70311bad73440 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:06.321432 16131 log.cc:1196] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/ecfefc1736ae4d5caade9e00af8a3718
I20240502 08:18:06.321830 16131 ts_tablet_manager.cc:1909] T ecfefc1736ae4d5caade9e00af8a3718 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:06.322252 15993 tablet_replica.cc:330] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:06.322407 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet bf1c03ab6bf840bf8fc70311bad73440 (table rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]) successfully deleted
I20240502 08:18:06.322515 15993 raft_consensus.cc:2229] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.322762 15993 raft_consensus.cc:2258] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.323431 15993 ts_tablet_manager.cc:1875] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.329480 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet ecfefc1736ae4d5caade9e00af8a3718 (table rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]) successfully deleted
I20240502 08:18:06.330653 15993 ts_tablet_manager.cc:1888] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.330844 15993 log.cc:1196] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/ecfefc1736ae4d5caade9e00af8a3718
I20240502 08:18:06.331342 15993 ts_tablet_manager.cc:1909] T ecfefc1736ae4d5caade9e00af8a3718 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:06.333527 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe55ec22eb14db4808a0fe2054aab93" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:06.333833 15488 raft_consensus.cc:2454] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:06.334106 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet ecfefc1736ae4d5caade9e00af8a3718 (table rebalance_test_table_8 [id=4958f06f59f54813b86cc3e5b7284dd3]) successfully deleted
I20240502 08:18:06.334528 12664 leader_election.cc:302] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:06.336412 16186 raft_consensus.cc:2781] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:06.336529 16186 raft_consensus.cc:482] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:06.336656 16186 raft_consensus.cc:3037] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:06.336786 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe55ec22eb14db4808a0fe2054aab93" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:06.337003 15755 raft_consensus.cc:2454] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:06.338892 16186 raft_consensus.cc:504] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:06.339427 16186 leader_election.cc:288] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:06.339820 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe55ec22eb14db4808a0fe2054aab93" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:06.339957 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe55ec22eb14db4808a0fe2054aab93" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:06.340032 15488 raft_consensus.cc:3037] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:06.340117 15755 raft_consensus.cc:3037] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:06.342795 15755 raft_consensus.cc:2454] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:06.342789 15488 raft_consensus.cc:2454] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:06.343201 12664 leader_election.cc:302] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:06.343451 16054 raft_consensus.cc:2781] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:06.343632 16054 raft_consensus.cc:686] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:18:06.343894 16054 consensus_queue.cc:237] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:06.347169 12584 catalog_manager.cc:5484] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:06.482173 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:06.487943 15755 raft_consensus.cc:1261] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:06.488046 15488 raft_consensus.cc:1261] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:06.488615 16186 consensus_queue.cc:1035] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 08:18:06.489475 16054 consensus_queue.cc:1035] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:06.492587 12586 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet dbe55ec22eb14db4808a0fe2054aab93 (table rebalancer_dynamic_table [id=56f4ff84513146ccb41ef079a6a6f11e]). Expected version 1 got 0
I20240502 08:18:06.493508 16193 tablet.cc:1722] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:06.493779 12586 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet dbe55ec22eb14db4808a0fe2054aab93 (table rebalancer_dynamic_table [id=56f4ff84513146ccb41ef079a6a6f11e]). Expected version 1 got 0
I20240502 08:18:06.494130 16193 tablet.cc:1620] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:06.496941 16195 tablet.cc:1722] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:06.497480 16195 tablet.cc:1620] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:06.498113 16194 tablet.cc:1722] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:06.498376 12584 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet dbe55ec22eb14db4808a0fe2054aab93 (table rebalancer_dynamic_table [id=56f4ff84513146ccb41ef079a6a6f11e]). Expected version 1 got 0
I20240502 08:18:06.499629 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet dbe55ec22eb14db4808a0fe2054aab93 (table rebalancer_dynamic_table [id=56f4ff84513146ccb41ef079a6a6f11e]). Expected version 1 got 0
I20240502 08:18:06.500157 16194 tablet.cc:1620] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:06.504717 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=56f4ff84513146ccb41ef079a6a6f11e] alter complete (version 1)
I20240502 08:18:06.606559 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:06.614432 16193 tablet.cc:1722] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:06.614940 16193 tablet.cc:1620] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:06.615797 16194 tablet.cc:1722] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:06.616385 16194 tablet.cc:1620] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:06.615010 16195 tablet.cc:1722] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:06.619359 16195 tablet.cc:1620] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:06.624082 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=56f4ff84513146ccb41ef079a6a6f11e] alter complete (version 2)
I20240502 08:18:06.727289 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:06.734364 16193 tablet.cc:1722] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:06.734745 16194 tablet.cc:1722] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:06.736193 16195 tablet.cc:1722] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:06.742944 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=56f4ff84513146ccb41ef079a6a6f11e] alter complete (version 3)
I20240502 08:18:06.784466 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 08:18:06.784700 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 08:18:06.788014 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet e506ca0472444a4686609f2c42d66345
I20240502 08:18:06.788514 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 6a7f5bb648c042a5b18d0a04b01ca6a7
I20240502 08:18:06.788690 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet e506ca0472444a4686609f2c42d66345 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:06.788758 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet e506ca0472444a4686609f2c42d66345 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:06.789005 15994 tablet_replica.cc:330] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:06.789011 15996 tablet_replica.cc:330] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:06.789178 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7f5bb648c042a5b18d0a04b01ca6a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:06.789232 15994 raft_consensus.cc:2229] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.789223 15996 raft_consensus.cc:2229] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.789439 15996 raft_consensus.cc:2258] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.789453 15994 raft_consensus.cc:2258] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.789458 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet e506ca0472444a4686609f2c42d66345 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:06.789561 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet f68020cc7c1346c888c35c0da7b71b94
I20240502 08:18:06.789733 15993 tablet_replica.cc:330] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:06.789911 15993 raft_consensus.cc:2229] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:06.790155 15996 ts_tablet_manager.cc:1875] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.790166 15994 ts_tablet_manager.cc:1875] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.790212 15993 raft_consensus.cc:2258] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.790885 16131 tablet_replica.cc:330] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:06.791088 16131 raft_consensus.cc:2229] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:06.791200 12972 tablet_service.cc:1514] Processing DeleteTablet for tablet f68020cc7c1346c888c35c0da7b71b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:06.791366 16131 raft_consensus.cc:2258] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.792016 16131 ts_tablet_manager.cc:1875] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.793877 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7f5bb648c042a5b18d0a04b01ca6a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:06.794100 12840 tablet_service.cc:1514] Processing DeleteTablet for tablet f68020cc7c1346c888c35c0da7b71b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:06.794782 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a7f5bb648c042a5b18d0a04b01ca6a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:06.795285 15996 ts_tablet_manager.cc:1888] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.795404 15996 log.cc:1196] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/e506ca0472444a4686609f2c42d66345
I20240502 08:18:06.795727 15996 ts_tablet_manager.cc:1909] T e506ca0472444a4686609f2c42d66345 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:06.796419 15996 tablet_replica.cc:330] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:06.796726 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet e506ca0472444a4686609f2c42d66345 (table rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]) successfully deleted
I20240502 08:18:06.797123 15996 raft_consensus.cc:2229] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.797099 15994 ts_tablet_manager.cc:1888] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.797328 15996 raft_consensus.cc:2258] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.797339 15994 log.cc:1196] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/e506ca0472444a4686609f2c42d66345
I20240502 08:18:06.797937 15996 ts_tablet_manager.cc:1875] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.798928 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet f68020cc7c1346c888c35c0da7b71b94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:06.799532 15994 ts_tablet_manager.cc:1909] T e506ca0472444a4686609f2c42d66345 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:06.799582 15993 ts_tablet_manager.cc:1875] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.800392 15994 tablet_replica.cc:330] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:06.800563 15994 raft_consensus.cc:2229] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:06.800869 15994 raft_consensus.cc:2258] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.801517 15994 ts_tablet_manager.cc:1875] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.802052 15996 ts_tablet_manager.cc:1888] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.802160 15996 log.cc:1196] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/6a7f5bb648c042a5b18d0a04b01ca6a7
I20240502 08:18:06.802485 15996 ts_tablet_manager.cc:1909] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:06.802805 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet e506ca0472444a4686609f2c42d66345 (table rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]) successfully deleted
I20240502 08:18:06.803267 15996 tablet_replica.cc:330] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:06.803907 15996 raft_consensus.cc:2229] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.804157 15996 raft_consensus.cc:2258] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.804313 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 6a7f5bb648c042a5b18d0a04b01ca6a7 (table rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]) successfully deleted
I20240502 08:18:06.804823 15996 ts_tablet_manager.cc:1875] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.806988 15994 ts_tablet_manager.cc:1888] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.807101 15994 log.cc:1196] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/f68020cc7c1346c888c35c0da7b71b94
I20240502 08:18:06.807514 15994 ts_tablet_manager.cc:1909] T f68020cc7c1346c888c35c0da7b71b94 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:06.807909 16131 ts_tablet_manager.cc:1888] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.808044 16131 log.cc:1196] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/e506ca0472444a4686609f2c42d66345
I20240502 08:18:06.808362 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet f68020cc7c1346c888c35c0da7b71b94 (table rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]) successfully deleted
I20240502 08:18:06.808425 16131 ts_tablet_manager.cc:1909] T e506ca0472444a4686609f2c42d66345 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:06.808758 15996 ts_tablet_manager.cc:1888] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.808879 15996 log.cc:1196] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/f68020cc7c1346c888c35c0da7b71b94
I20240502 08:18:06.809108 16131 tablet_replica.cc:330] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:06.809176 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet e506ca0472444a4686609f2c42d66345 (table rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]) successfully deleted
I20240502 08:18:06.809243 15996 ts_tablet_manager.cc:1909] T f68020cc7c1346c888c35c0da7b71b94 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:06.809309 16131 raft_consensus.cc:2229] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.809521 16131 raft_consensus.cc:2258] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.810051 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet f68020cc7c1346c888c35c0da7b71b94 (table rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]) successfully deleted
I20240502 08:18:06.813313 16131 ts_tablet_manager.cc:1875] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.813544 15993 ts_tablet_manager.cc:1888] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.813654 15993 log.cc:1196] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/6a7f5bb648c042a5b18d0a04b01ca6a7
I20240502 08:18:06.814045 15993 ts_tablet_manager.cc:1909] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:06.815021 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 6a7f5bb648c042a5b18d0a04b01ca6a7 (table rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]) successfully deleted
I20240502 08:18:06.817584 16131 ts_tablet_manager.cc:1888] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.817682 16131 log.cc:1196] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/f68020cc7c1346c888c35c0da7b71b94
I20240502 08:18:06.817992 16131 ts_tablet_manager.cc:1909] T f68020cc7c1346c888c35c0da7b71b94 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:06.818657 16131 tablet_replica.cc:330] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:06.818732 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet f68020cc7c1346c888c35c0da7b71b94 (table rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]) successfully deleted
I20240502 08:18:06.818838 16131 raft_consensus.cc:2229] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.819036 16131 raft_consensus.cc:2258] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.819566 16131 ts_tablet_manager.cc:1875] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.823668 16131 ts_tablet_manager.cc:1888] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:06.823782 16131 log.cc:1196] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/6a7f5bb648c042a5b18d0a04b01ca6a7
I20240502 08:18:06.824061 16131 ts_tablet_manager.cc:1909] T 6a7f5bb648c042a5b18d0a04b01ca6a7 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:06.824860 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 6a7f5bb648c042a5b18d0a04b01ca6a7 (table rebalance_test_table_10 [id=906b31cb74364f30b829cd3e342b6939]) successfully deleted
I20240502 08:18:06.844897 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:06.845055 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:06.847774 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet dbe55ec22eb14db4808a0fe2054aab93
I20240502 08:18:06.848538 15734 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe55ec22eb14db4808a0fe2054aab93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:06.848801 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe55ec22eb14db4808a0fe2054aab93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:06.848912 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe55ec22eb14db4808a0fe2054aab93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:06 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:06.849169 16196 tablet_replica.cc:330] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:06.849417 15994 tablet_replica.cc:330] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:06.849457 16197 tablet_replica.cc:330] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:06.849465 16196 raft_consensus.cc:2229] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.849661 16196 raft_consensus.cc:2258] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.849697 16197 raft_consensus.cc:2229] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:06.849774 15994 raft_consensus.cc:2229] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:06.850147 15994 raft_consensus.cc:2258] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.850575 16196 ts_tablet_manager.cc:1875] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.851027 15994 ts_tablet_manager.cc:1875] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.853049 16197 raft_consensus.cc:2258] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:06.853864 16197 ts_tablet_manager.cc:1875] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:06.854975 16196 ts_tablet_manager.cc:1888] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:06.855132 16196 log.cc:1196] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/dbe55ec22eb14db4808a0fe2054aab93
I20240502 08:18:06.855573 16196 ts_tablet_manager.cc:1909] T dbe55ec22eb14db4808a0fe2054aab93 P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:06.856150 15994 ts_tablet_manager.cc:1888] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:06.856357 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet dbe55ec22eb14db4808a0fe2054aab93 (table rebalancer_dynamic_table_new_name [id=56f4ff84513146ccb41ef079a6a6f11e]) successfully deleted
I20240502 08:18:06.856638 15994 log.cc:1196] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/dbe55ec22eb14db4808a0fe2054aab93
I20240502 08:18:06.857009 15994 ts_tablet_manager.cc:1909] T dbe55ec22eb14db4808a0fe2054aab93 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:06.857939 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet dbe55ec22eb14db4808a0fe2054aab93 (table rebalancer_dynamic_table_new_name [id=56f4ff84513146ccb41ef079a6a6f11e]) successfully deleted
I20240502 08:18:06.859216 16197 ts_tablet_manager.cc:1888] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:06.859311 16197 log.cc:1196] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/dbe55ec22eb14db4808a0fe2054aab93
I20240502 08:18:06.859629 16197 ts_tablet_manager.cc:1909] T dbe55ec22eb14db4808a0fe2054aab93 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:06.860373 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet dbe55ec22eb14db4808a0fe2054aab93 (table rebalancer_dynamic_table_new_name [id=56f4ff84513146ccb41ef079a6a6f11e]) successfully deleted
I20240502 08:18:06.950084 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:06.956894 15867 tablet_service.cc:1467] Processing CreateTablet for tablet ce3ff81875ee4211b86704aa01b444a5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ca38144857f448ab81422d99154e378]), partition=
I20240502 08:18:06.957113 15467 tablet_service.cc:1467] Processing CreateTablet for tablet ce3ff81875ee4211b86704aa01b444a5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ca38144857f448ab81422d99154e378]), partition=
I20240502 08:18:06.957273 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3ff81875ee4211b86704aa01b444a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.957324 12709 tablet_service.cc:1467] Processing CreateTablet for tablet ce3ff81875ee4211b86704aa01b444a5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ca38144857f448ab81422d99154e378]), partition=
I20240502 08:18:06.957464 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3ff81875ee4211b86704aa01b444a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.957674 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3ff81875ee4211b86704aa01b444a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.962359 16198 tablet_bootstrap.cc:492] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:06.964337 16198 tablet_bootstrap.cc:654] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.964495 16199 tablet_bootstrap.cc:492] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:06.966172 16199 tablet_bootstrap.cc:654] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.967057 16198 tablet_bootstrap.cc:492] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:06.967240 16198 ts_tablet_manager.cc:1367] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 08:18:06.967533 16201 tablet_bootstrap.cc:492] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:06.968102 16198 raft_consensus.cc:348] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:06.968416 16198 raft_consensus.cc:374] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.968504 16198 raft_consensus.cc:729] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.969015 16198 consensus_queue.cc:260] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:06.969288 16199 tablet_bootstrap.cc:492] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:06.969465 16199 ts_tablet_manager.cc:1367] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 08:18:06.969683 16201 tablet_bootstrap.cc:654] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.970235 16199 raft_consensus.cc:348] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:06.970515 16199 raft_consensus.cc:374] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.970651 16199 raft_consensus.cc:729] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.970888 16199 consensus_queue.cc:260] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:06.970611 16198 ts_tablet_manager.cc:1398] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:18:06.971485 16199 ts_tablet_manager.cc:1398] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:06.975359 16201 tablet_bootstrap.cc:492] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:06.975484 16201 ts_tablet_manager.cc:1367] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 08:18:06.976052 16201 raft_consensus.cc:348] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:06.976253 16201 raft_consensus.cc:374] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.976344 16201 raft_consensus.cc:729] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:06.977710 16201 consensus_queue.cc:260] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:06.978264 16201 ts_tablet_manager.cc:1398] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:07.125077 16057 raft_consensus.cc:482] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:07.125253 16057 raft_consensus.cc:504] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:07.125844 16057 leader_election.cc:288] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:07.126252 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:07.126283 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:07.126498 15887 raft_consensus.cc:2454] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:07.126515 15622 raft_consensus.cc:2454] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:07.126976 15689 leader_election.cc:302] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:07.127272 16057 raft_consensus.cc:2781] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:18:07.127410 16057 raft_consensus.cc:482] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:07.127501 16057 raft_consensus.cc:3037] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:07.131163 16057 raft_consensus.cc:504] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:07.131918 16057 leader_election.cc:288] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:07.132272 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:07.132481 15887 raft_consensus.cc:3037] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:07.132443 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040878c4c4ae4684b02296ee9eac8985" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:07.132623 15622 raft_consensus.cc:3037] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:07.135149 15887 raft_consensus.cc:2454] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 2.
I20240502 08:18:07.135160 15622 raft_consensus.cc:2454] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 2.
I20240502 08:18:07.135558 15690 leader_election.cc:302] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:07.135890 16057 raft_consensus.cc:2781] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:07.145279 16057 raft_consensus.cc:686] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 2 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:07.145681 16057 consensus_queue.cc:237] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:07.148511 12579 catalog_manager.cc:5484] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 2, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 2 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:07.300211 12579 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 08:18:07.300427 12579 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 08:18:07.303915 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet e10246a0daff4378a48013a25eb17c82
I20240502 08:18:07.304507 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 214793210c6c429b86fd2351c7e52325
I20240502 08:18:07.304581 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet e10246a0daff4378a48013a25eb17c82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:07.304868 15994 tablet_replica.cc:330] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:07.304906 12840 tablet_service.cc:1514] Processing DeleteTablet for tablet e10246a0daff4378a48013a25eb17c82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:07.304984 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 3c3b998826974f4e9ba61021af981de9
I20240502 08:18:07.305058 15994 raft_consensus.cc:2229] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.305209 15996 tablet_replica.cc:330] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:07.305179 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet e10246a0daff4378a48013a25eb17c82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:07.305299 15994 raft_consensus.cc:2258] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.305392 15996 raft_consensus.cc:2229] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.305460 15993 tablet_replica.cc:330] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:07.305651 15993 raft_consensus.cc:2229] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:07.305653 15996 raft_consensus.cc:2258] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.305835 12840 tablet_service.cc:1514] Processing DeleteTablet for tablet 214793210c6c429b86fd2351c7e52325 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:07.305940 15993 raft_consensus.cc:2258] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.305984 15994 ts_tablet_manager.cc:1875] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.306329 15996 ts_tablet_manager.cc:1875] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.306622 15993 ts_tablet_manager.cc:1875] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.306628 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 214793210c6c429b86fd2351c7e52325 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:07.310606 16131 tablet_replica.cc:330] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:07.310793 16131 raft_consensus.cc:2229] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.311043 16131 raft_consensus.cc:2258] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.311573 16131 ts_tablet_manager.cc:1875] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.312018 15994 ts_tablet_manager.cc:1888] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.312136 15994 log.cc:1196] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/e10246a0daff4378a48013a25eb17c82
I20240502 08:18:07.312476 15993 ts_tablet_manager.cc:1888] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.312551 15994 ts_tablet_manager.cc:1909] T e10246a0daff4378a48013a25eb17c82 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:07.312611 15993 log.cc:1196] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/e10246a0daff4378a48013a25eb17c82
I20240502 08:18:07.312973 15993 ts_tablet_manager.cc:1909] T e10246a0daff4378a48013a25eb17c82 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:07.313500 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet 214793210c6c429b86fd2351c7e52325 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:07.313784 15994 tablet_replica.cc:330] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:07.314025 15994 raft_consensus.cc:2229] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.314277 15994 raft_consensus.cc:2258] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.314592 16024 raft_consensus.cc:482] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:07.314752 16024 raft_consensus.cc:504] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:07.315073 15994 ts_tablet_manager.cc:1875] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.315414 15996 ts_tablet_manager.cc:1888] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.315539 15996 log.cc:1196] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/e10246a0daff4378a48013a25eb17c82
I20240502 08:18:07.315532 16024 leader_election.cc:288] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:07.315976 15996 ts_tablet_manager.cc:1909] T e10246a0daff4378a48013a25eb17c82 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:07.316741 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3ff81875ee4211b86704aa01b444a5" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:07.316998 15996 tablet_replica.cc:330] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:07.316988 15488 raft_consensus.cc:2454] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:07.317169 15996 raft_consensus.cc:2229] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:07.317466 15996 raft_consensus.cc:2258] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.318286 15996 ts_tablet_manager.cc:1875] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.319021 16131 ts_tablet_manager.cc:1888] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.319133 16131 log.cc:1196] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/214793210c6c429b86fd2351c7e52325
I20240502 08:18:07.319484 16131 ts_tablet_manager.cc:1909] T 214793210c6c429b86fd2351c7e52325 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:07.322614 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet e10246a0daff4378a48013a25eb17c82 (table rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]) successfully deleted
I20240502 08:18:07.322821 15994 ts_tablet_manager.cc:1888] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.322872 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet e10246a0daff4378a48013a25eb17c82 (table rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]) successfully deleted
I20240502 08:18:07.323067 15994 log.cc:1196] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/214793210c6c429b86fd2351c7e52325
I20240502 08:18:07.323695 12840 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c3b998826974f4e9ba61021af981de9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:07.324083 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet e10246a0daff4378a48013a25eb17c82 (table rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]) successfully deleted
I20240502 08:18:07.324326 15994 ts_tablet_manager.cc:1909] T 214793210c6c429b86fd2351c7e52325 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:07.324976 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c3b998826974f4e9ba61021af981de9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:07.325280 15996 ts_tablet_manager.cc:1888] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.325280 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 214793210c6c429b86fd2351c7e52325 (table rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]) successfully deleted
I20240502 08:18:07.325410 15996 log.cc:1196] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/214793210c6c429b86fd2351c7e52325
I20240502 08:18:07.325876 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c3b998826974f4e9ba61021af981de9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:07.326148 15993 tablet_replica.cc:330] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:07.326133 15819 leader_election.cc:302] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:07.326332 15993 raft_consensus.cc:2229] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.326414 16024 raft_consensus.cc:2781] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:07.326582 15993 raft_consensus.cc:2258] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.326673 16024 raft_consensus.cc:482] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:07.326762 16024 raft_consensus.cc:3037] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.327176 15993 ts_tablet_manager.cc:1875] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.329519 16024 raft_consensus.cc:504] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:07.330614 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3ff81875ee4211b86704aa01b444a5" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:07.330854 15488 raft_consensus.cc:3037] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.332203 16131 tablet_replica.cc:330] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:07.332417 16131 raft_consensus.cc:2229] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:07.332716 16131 raft_consensus.cc:2258] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.333346 15993 ts_tablet_manager.cc:1888] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.333412 16131 ts_tablet_manager.cc:1875] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.333554 15993 log.cc:1196] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/3c3b998826974f4e9ba61021af981de9
I20240502 08:18:07.333703 15996 ts_tablet_manager.cc:1909] T 214793210c6c429b86fd2351c7e52325 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:07.333783 15488 raft_consensus.cc:2454] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:07.334492 15819 leader_election.cc:302] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:07.334950 15993 ts_tablet_manager.cc:1909] T 3c3b998826974f4e9ba61021af981de9 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:07.335836 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 3c3b998826974f4e9ba61021af981de9 (table rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]) successfully deleted
I20240502 08:18:07.336045 15996 tablet_replica.cc:330] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:07.336074 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 214793210c6c429b86fd2351c7e52325 (table rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]) successfully deleted
I20240502 08:18:07.336320 15996 raft_consensus.cc:2229] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.336606 15996 raft_consensus.cc:2258] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.337797 15996 ts_tablet_manager.cc:1875] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.343840 15996 ts_tablet_manager.cc:1888] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.343997 15996 log.cc:1196] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/3c3b998826974f4e9ba61021af981de9
I20240502 08:18:07.344166 16024 leader_election.cc:288] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:07.344388 16024 raft_consensus.cc:2781] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:07.344521 16024 raft_consensus.cc:686] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:07.344631 15996 ts_tablet_manager.cc:1909] T 3c3b998826974f4e9ba61021af981de9 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:07.344784 16024 consensus_queue.cc:237] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:07.345017 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 214793210c6c429b86fd2351c7e52325 (table rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]) successfully deleted
I20240502 08:18:07.345790 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 3c3b998826974f4e9ba61021af981de9 (table rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]) successfully deleted
I20240502 08:18:07.346941 16131 ts_tablet_manager.cc:1888] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.347127 16131 log.cc:1196] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/3c3b998826974f4e9ba61021af981de9
I20240502 08:18:07.347540 16131 ts_tablet_manager.cc:1909] T 3c3b998826974f4e9ba61021af981de9 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:07.348543 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 3c3b998826974f4e9ba61021af981de9 (table rebalance_test_table_12 [id=e5940738e82446fe97d69fd832e2e4ce]) successfully deleted
I20240502 08:18:07.354427 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3ff81875ee4211b86704aa01b444a5" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:07.354561 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3ff81875ee4211b86704aa01b444a5" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:07.354713 12730 raft_consensus.cc:2454] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:07.356259 12579 catalog_manager.cc:5484] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:07.524492 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:07.530074 12730 raft_consensus.cc:3037] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.530310 15488 raft_consensus.cc:1261] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:07.530978 16210 consensus_queue.cc:1035] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.532938 12730 raft_consensus.cc:1261] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:07.533669 16210 consensus_queue.cc:1035] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.535890 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet ce3ff81875ee4211b86704aa01b444a5 (table rebalancer_dynamic_table [id=5ca38144857f448ab81422d99154e378]). Expected version 1 got 0
I20240502 08:18:07.540040 12579 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet ce3ff81875ee4211b86704aa01b444a5 (table rebalancer_dynamic_table [id=5ca38144857f448ab81422d99154e378]). Expected version 1 got 0
I20240502 08:18:07.540237 16220 tablet.cc:1722] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:07.541294 16220 tablet.cc:1620] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:07.547083 16223 tablet.cc:1722] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:07.547780 16223 tablet.cc:1620] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:07.549057 12584 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet ce3ff81875ee4211b86704aa01b444a5 (table rebalancer_dynamic_table [id=5ca38144857f448ab81422d99154e378]). Expected version 1 got 0
I20240502 08:18:07.550500 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=5ca38144857f448ab81422d99154e378] alter complete (version 1)
I20240502 08:18:07.553102 16222 tablet.cc:1722] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:07.553529 16222 tablet.cc:1620] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:07.555312 12579 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet ce3ff81875ee4211b86704aa01b444a5 (table rebalancer_dynamic_table [id=5ca38144857f448ab81422d99154e378]). Expected version 1 got 0
I20240502 08:18:07.645937 15622 raft_consensus.cc:1261] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 2 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:18:07.646495 16057 consensus_queue.cc:1035] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.652719 15887 raft_consensus.cc:1261] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 08:18:07.653136 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:07.653328 16228 consensus_queue.cc:1035] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.660826 16220 tablet.cc:1722] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:07.661293 16220 tablet.cc:1620] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:07.661597 16222 tablet.cc:1722] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:07.662040 16222 tablet.cc:1620] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:07.666116 16223 tablet.cc:1722] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:07.666563 16223 tablet.cc:1620] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:07.676537 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=5ca38144857f448ab81422d99154e378] alter complete (version 2)
I20240502 08:18:07.778508 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:07.785682 16220 tablet.cc:1722] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:07.786424 16222 tablet.cc:1722] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:07.786592 16223 tablet.cc:1722] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:07.793467 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=5ca38144857f448ab81422d99154e378] alter complete (version 3)
I20240502 08:18:07.824492 12579 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 08:18:07.824726 12579 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 08:18:07.827780 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 22b69a35c6c6413fb3631c657cfd97f0
I20240502 08:18:07.828177 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 015aca35bbdf4fb09c8a801d5786522c
I20240502 08:18:07.828456 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b69a35c6c6413fb3631c657cfd97f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:07.828539 12840 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b69a35c6c6413fb3631c657cfd97f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:07.828743 16131 tablet_replica.cc:330] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:07.828787 15996 tablet_replica.cc:330] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:07.828930 16131 raft_consensus.cc:2229] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.828998 12840 tablet_service.cc:1514] Processing DeleteTablet for tablet 015aca35bbdf4fb09c8a801d5786522c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:07.829022 15996 raft_consensus.cc:2229] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.829169 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b69a35c6c6413fb3631c657cfd97f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:07.829234 16131 raft_consensus.cc:2258] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.829234 15996 raft_consensus.cc:2258] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.829732 16230 tablet_replica.cc:330] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:07.829912 15996 ts_tablet_manager.cc:1875] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.829928 16131 ts_tablet_manager.cc:1875] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.829957 16230 raft_consensus.cc:2229] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:07.830359 16230 raft_consensus.cc:2258] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.831041 16230 ts_tablet_manager.cc:1875] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.831477 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 015aca35bbdf4fb09c8a801d5786522c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:07.832351 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet c4438de6e9f14717acba7c9cf279bb4e
I20240502 08:18:07.832412 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 015aca35bbdf4fb09c8a801d5786522c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:07.832661 15993 tablet_replica.cc:330] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:07.832863 15993 raft_consensus.cc:2229] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:07.832950 12840 tablet_service.cc:1514] Processing DeleteTablet for tablet c4438de6e9f14717acba7c9cf279bb4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:07.833169 15993 raft_consensus.cc:2258] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.833313 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet c4438de6e9f14717acba7c9cf279bb4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:07.833874 15993 ts_tablet_manager.cc:1875] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.833972 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet c4438de6e9f14717acba7c9cf279bb4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:07.836370 16131 ts_tablet_manager.cc:1888] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.836473 16131 log.cc:1196] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/22b69a35c6c6413fb3631c657cfd97f0
I20240502 08:18:07.836956 16131 ts_tablet_manager.cc:1909] T 22b69a35c6c6413fb3631c657cfd97f0 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:07.837776 16131 tablet_replica.cc:330] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:07.837857 16230 ts_tablet_manager.cc:1888] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.837888 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 22b69a35c6c6413fb3631c657cfd97f0 (table rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]) successfully deleted
I20240502 08:18:07.837958 16230 log.cc:1196] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/22b69a35c6c6413fb3631c657cfd97f0
I20240502 08:18:07.837980 16131 raft_consensus.cc:2229] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.838151 16131 raft_consensus.cc:2258] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.838379 16230 ts_tablet_manager.cc:1909] T 22b69a35c6c6413fb3631c657cfd97f0 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:07.838644 16131 ts_tablet_manager.cc:1875] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.839352 16230 tablet_replica.cc:330] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:07.839470 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 22b69a35c6c6413fb3631c657cfd97f0 (table rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]) successfully deleted
I20240502 08:18:07.839648 16230 raft_consensus.cc:2229] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.839897 16230 raft_consensus.cc:2258] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.840667 16230 ts_tablet_manager.cc:1875] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.842823 16131 ts_tablet_manager.cc:1888] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.842984 16131 log.cc:1196] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/015aca35bbdf4fb09c8a801d5786522c
I20240502 08:18:07.843292 16131 ts_tablet_manager.cc:1909] T 015aca35bbdf4fb09c8a801d5786522c P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:07.843809 15996 ts_tablet_manager.cc:1888] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.843963 15996 log.cc:1196] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/22b69a35c6c6413fb3631c657cfd97f0
I20240502 08:18:07.844081 16131 tablet_replica.cc:330] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:07.844201 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 015aca35bbdf4fb09c8a801d5786522c (table rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]) successfully deleted
I20240502 08:18:07.844357 16131 raft_consensus.cc:2229] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.844419 15996 ts_tablet_manager.cc:1909] T 22b69a35c6c6413fb3631c657cfd97f0 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:07.844599 16131 raft_consensus.cc:2258] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.845111 16131 ts_tablet_manager.cc:1875] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.845389 15996 tablet_replica.cc:330] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:07.845664 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 22b69a35c6c6413fb3631c657cfd97f0 (table rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]) successfully deleted
I20240502 08:18:07.846977 16230 ts_tablet_manager.cc:1888] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.847187 16230 log.cc:1196] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/c4438de6e9f14717acba7c9cf279bb4e
I20240502 08:18:07.847723 16230 ts_tablet_manager.cc:1909] T c4438de6e9f14717acba7c9cf279bb4e P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:07.848152 15993 ts_tablet_manager.cc:1888] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.848357 15993 log.cc:1196] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/015aca35bbdf4fb09c8a801d5786522c
I20240502 08:18:07.848886 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet c4438de6e9f14717acba7c9cf279bb4e (table rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]) successfully deleted
I20240502 08:18:07.848964 15993 ts_tablet_manager.cc:1909] T 015aca35bbdf4fb09c8a801d5786522c P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:07.849786 15996 raft_consensus.cc:2229] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.849865 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 015aca35bbdf4fb09c8a801d5786522c (table rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]) successfully deleted
I20240502 08:18:07.849972 16131 ts_tablet_manager.cc:1888] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.850013 15996 raft_consensus.cc:2258] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.850075 16131 log.cc:1196] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/c4438de6e9f14717acba7c9cf279bb4e
I20240502 08:18:07.850661 16131 ts_tablet_manager.cc:1909] T c4438de6e9f14717acba7c9cf279bb4e P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:07.850634 15996 ts_tablet_manager.cc:1875] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.851562 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet c4438de6e9f14717acba7c9cf279bb4e (table rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]) successfully deleted
I20240502 08:18:07.855175 15996 ts_tablet_manager.cc:1888] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.855309 15996 log.cc:1196] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/015aca35bbdf4fb09c8a801d5786522c
I20240502 08:18:07.855660 15996 ts_tablet_manager.cc:1909] T 015aca35bbdf4fb09c8a801d5786522c P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:07.856339 15996 tablet_replica.cc:330] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:07.856403 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 015aca35bbdf4fb09c8a801d5786522c (table rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]) successfully deleted
I20240502 08:18:07.856530 15996 raft_consensus.cc:2229] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:07.856792 15996 raft_consensus.cc:2258] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.857324 15996 ts_tablet_manager.cc:1875] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.861063 15996 ts_tablet_manager.cc:1888] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:07.861193 15996 log.cc:1196] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/c4438de6e9f14717acba7c9cf279bb4e
I20240502 08:18:07.861537 15996 ts_tablet_manager.cc:1909] T c4438de6e9f14717acba7c9cf279bb4e P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:07.862301 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet c4438de6e9f14717acba7c9cf279bb4e (table rebalance_test_table_14 [id=906ea57c8914429b97804fb9aac6aa4d]) successfully deleted
I20240502 08:18:07.895685 12579 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:07.895895 12579 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:07.898707 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet ce3ff81875ee4211b86704aa01b444a5
I20240502 08:18:07.899534 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3ff81875ee4211b86704aa01b444a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:07.899581 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3ff81875ee4211b86704aa01b444a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:07.899899 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3ff81875ee4211b86704aa01b444a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:07 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:07.900161 16231 tablet_replica.cc:330] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:07.900290 16230 tablet_replica.cc:330] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:07.900533 16232 tablet_replica.cc:330] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:07.900559 16230 raft_consensus.cc:2229] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.900570 16231 raft_consensus.cc:2229] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:07.900712 16232 raft_consensus.cc:2229] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:07.900806 16230 raft_consensus.cc:2258] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.900977 16232 raft_consensus.cc:2258] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.901067 16231 raft_consensus.cc:2258] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:07.901983 16231 ts_tablet_manager.cc:1875] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.901983 16230 ts_tablet_manager.cc:1875] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.906942 16232 ts_tablet_manager.cc:1875] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:07.908133 16231 ts_tablet_manager.cc:1888] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:07.908133 16230 ts_tablet_manager.cc:1888] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:07.908253 16231 log.cc:1196] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/ce3ff81875ee4211b86704aa01b444a5
I20240502 08:18:07.908257 16230 log.cc:1196] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/ce3ff81875ee4211b86704aa01b444a5
I20240502 08:18:07.908669 16230 ts_tablet_manager.cc:1909] T ce3ff81875ee4211b86704aa01b444a5 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:07.908680 16231 ts_tablet_manager.cc:1909] T ce3ff81875ee4211b86704aa01b444a5 P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:07.909644 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet ce3ff81875ee4211b86704aa01b444a5 (table rebalancer_dynamic_table_new_name [id=5ca38144857f448ab81422d99154e378]) successfully deleted
I20240502 08:18:07.909705 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet ce3ff81875ee4211b86704aa01b444a5 (table rebalancer_dynamic_table_new_name [id=5ca38144857f448ab81422d99154e378]) successfully deleted
I20240502 08:18:07.911366 16232 ts_tablet_manager.cc:1888] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:07.911463 16232 log.cc:1196] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/ce3ff81875ee4211b86704aa01b444a5
I20240502 08:18:07.911882 16232 ts_tablet_manager.cc:1909] T ce3ff81875ee4211b86704aa01b444a5 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:07.912691 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet ce3ff81875ee4211b86704aa01b444a5 (table rebalancer_dynamic_table_new_name [id=5ca38144857f448ab81422d99154e378]) successfully deleted
I20240502 08:18:07.991956 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:07.999874 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab3748794114768bba90e903384819c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c68337015ae744879e3d0cbe8455502e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:07.999905 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab3748794114768bba90e903384819c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c68337015ae744879e3d0cbe8455502e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.000308 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab3748794114768bba90e903384819c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.000337 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab3748794114768bba90e903384819c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.001463 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:08.001543 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 3c58721561064e97909a0caddc47a1f0 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c68337015ae744879e3d0cbe8455502e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.001974 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c58721561064e97909a0caddc47a1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.002820 15466 tablet_service.cc:1467] Processing CreateTablet for tablet 3c58721561064e97909a0caddc47a1f0 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c68337015ae744879e3d0cbe8455502e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.003157 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c58721561064e97909a0caddc47a1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.006137 16233 tablet_bootstrap.cc:492] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:08.007230 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab3748794114768bba90e903384819c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c68337015ae744879e3d0cbe8455502e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.007215 15734 tablet_service.cc:1467] Processing CreateTablet for tablet a542eb208b254055b02765aceba5b405 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c68337015ae744879e3d0cbe8455502e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.007477 15867 tablet_service.cc:1467] Processing CreateTablet for tablet a542eb208b254055b02765aceba5b405 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c68337015ae744879e3d0cbe8455502e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.007609 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab3748794114768bba90e903384819c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.007644 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a542eb208b254055b02765aceba5b405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.007885 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a542eb208b254055b02765aceba5b405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.007973 16233 tablet_bootstrap.cc:654] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.010521 15866 tablet_service.cc:1467] Processing CreateTablet for tablet 3c58721561064e97909a0caddc47a1f0 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c68337015ae744879e3d0cbe8455502e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.010900 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c58721561064e97909a0caddc47a1f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.015477 12709 tablet_service.cc:1467] Processing CreateTablet for tablet a542eb208b254055b02765aceba5b405 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=c68337015ae744879e3d0cbe8455502e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.015898 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a542eb208b254055b02765aceba5b405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.023517 15865 tablet_service.cc:1467] Processing CreateTablet for tablet e9468dba95134fb4be53029dc313de7d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dee6b2bb26f2495bacf2c94f8ca92ad4]), partition=
I20240502 08:18:08.023835 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9468dba95134fb4be53029dc313de7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.025200 12841 tablet_service.cc:1467] Processing CreateTablet for tablet e9468dba95134fb4be53029dc313de7d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dee6b2bb26f2495bacf2c94f8ca92ad4]), partition=
I20240502 08:18:08.025622 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9468dba95134fb4be53029dc313de7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.031730 15467 tablet_service.cc:1467] Processing CreateTablet for tablet e9468dba95134fb4be53029dc313de7d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dee6b2bb26f2495bacf2c94f8ca92ad4]), partition=
I20240502 08:18:08.032121 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9468dba95134fb4be53029dc313de7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.042346 16236 tablet_bootstrap.cc:492] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:08.042800 16237 tablet_bootstrap.cc:492] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:08.044876 16236 tablet_bootstrap.cc:654] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.046917 16237 tablet_bootstrap.cc:654] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.052428 16237 tablet_bootstrap.cc:492] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:08.053048 16237 ts_tablet_manager.cc:1367] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 08:18:08.053977 16237 raft_consensus.cc:348] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.055651 16237 raft_consensus.cc:374] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.055917 16237 raft_consensus.cc:729] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.056404 16237 consensus_queue.cc:260] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.056608 16233 tablet_bootstrap.cc:492] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:08.056821 16233 ts_tablet_manager.cc:1367] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.051s user 0.025s sys 0.008s
I20240502 08:18:08.057624 16233 raft_consensus.cc:348] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:08.057888 16233 raft_consensus.cc:374] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.058027 16233 raft_consensus.cc:729] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.058548 16233 consensus_queue.cc:260] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:08.059006 16237 ts_tablet_manager.cc:1398] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 08:18:08.059710 16237 tablet_bootstrap.cc:492] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:08.061373 16237 tablet_bootstrap.cc:654] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.061478 16233 ts_tablet_manager.cc:1398] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240502 08:18:08.061805 16233 tablet_bootstrap.cc:492] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:08.063472 16233 tablet_bootstrap.cc:654] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.069084 16237 tablet_bootstrap.cc:492] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:08.069809 16237 ts_tablet_manager.cc:1367] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 08:18:08.071947 16237 raft_consensus.cc:348] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:08.072727 16237 raft_consensus.cc:374] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.072999 16237 raft_consensus.cc:729] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.073526 16237 consensus_queue.cc:260] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:08.075814 16237 ts_tablet_manager.cc:1398] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:08.080094 16237 tablet_bootstrap.cc:492] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:08.085194 16237 tablet_bootstrap.cc:654] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.094600 16246 tablet_bootstrap.cc:492] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:08.097581 16237 tablet_bootstrap.cc:492] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:08.098177 16237 ts_tablet_manager.cc:1367] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240502 08:18:08.101519 16237 raft_consensus.cc:348] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:08.102221 16246 tablet_bootstrap.cc:654] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.103382 16237 raft_consensus.cc:374] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.103660 16237 raft_consensus.cc:729] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.104045 16237 consensus_queue.cc:260] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:08.108650 16237 ts_tablet_manager.cc:1398] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:18:08.114352 16240 tablet_bootstrap.cc:492] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:08.115418 16246 tablet_bootstrap.cc:492] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:08.115986 16246 ts_tablet_manager.cc:1367] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240502 08:18:08.116220 16240 tablet_bootstrap.cc:654] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.118631 16247 tablet_bootstrap.cc:492] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:08.118932 16242 raft_consensus.cc:482] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.118886 16246 raft_consensus.cc:348] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:08.119100 16246 raft_consensus.cc:374] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.119179 16246 raft_consensus.cc:729] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.119074 16242 raft_consensus.cc:504] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:08.119988 16242 leader_election.cc:288] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:08.122164 16235 tablet_bootstrap.cc:492] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:08.122054 16246 consensus_queue.cc:260] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:08.123600 16235 tablet_bootstrap.cc:654] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.123720 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab3748794114768bba90e903384819c" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
W20240502 08:18:08.124481 15419 leader_election.cc:341] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:08.125056 16247 tablet_bootstrap.cc:654] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.127444 16247 tablet_bootstrap.cc:492] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:08.127905 16247 ts_tablet_manager.cc:1367] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:18:08.128337 16240 tablet_bootstrap.cc:492] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:08.128520 16240 ts_tablet_manager.cc:1367] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 08:18:08.128787 16247 raft_consensus.cc:348] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:08.129011 16247 raft_consensus.cc:374] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.129144 16247 raft_consensus.cc:729] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.129385 16247 consensus_queue.cc:260] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:08.129369 16240 raft_consensus.cc:348] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:08.129606 16240 raft_consensus.cc:374] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.129750 16240 raft_consensus.cc:729] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.130031 16240 consensus_queue.cc:260] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:08.130887 16247 ts_tablet_manager.cc:1398] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:08.132288 16246 ts_tablet_manager.cc:1398] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240502 08:18:08.134358 16221 raft_consensus.cc:482] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.134539 16221 raft_consensus.cc:504] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:08.135373 16221 leader_election.cc:288] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:08.137596 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9468dba95134fb4be53029dc313de7d" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
W20240502 08:18:08.138325 15819 leader_election.cc:341] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:08.138621 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab3748794114768bba90e903384819c" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:08.138840 13121 raft_consensus.cc:2454] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:08.139273 15418 leader_election.cc:302] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, db7893bb29a047b3b82cc60bfd8930a3; no voters: e1dd14f7ca764a1490df419f13f2804a
I20240502 08:18:08.139751 16242 raft_consensus.cc:2781] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:08.139871 16242 raft_consensus.cc:482] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:08.139966 16242 raft_consensus.cc:3037] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.142772 16242 raft_consensus.cc:504] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:08.143705 16242 leader_election.cc:288] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:08.143860 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab3748794114768bba90e903384819c" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:08.144032 13121 raft_consensus.cc:3037] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.146634 13121 raft_consensus.cc:2454] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:08.147024 15418 leader_election.cc:302] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:08.147349 16242 raft_consensus.cc:2781] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:08.147797 16242 raft_consensus.cc:686] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:08.148119 16242 consensus_queue.cc:237] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:08.150976 12584 catalog_manager.cc:5484] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:08.152245 16235 tablet_bootstrap.cc:492] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:08.152397 16235 ts_tablet_manager.cc:1367] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240502 08:18:08.157301 16235 raft_consensus.cc:348] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.157545 16235 raft_consensus.cc:374] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.157682 16235 raft_consensus.cc:729] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.157985 16235 consensus_queue.cc:260] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.158607 16235 ts_tablet_manager.cc:1398] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:18:08.167426 16240 ts_tablet_manager.cc:1398] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240502 08:18:08.167883 16233 tablet_bootstrap.cc:492] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:08.168071 16233 ts_tablet_manager.cc:1367] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.106s user 0.014s sys 0.006s
I20240502 08:18:08.168622 16240 tablet_bootstrap.cc:492] T 8ab3748794114768bba90e903384819c P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:08.168864 16233 raft_consensus.cc:348] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:08.169070 16233 raft_consensus.cc:374] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.169137 16233 raft_consensus.cc:729] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.169371 16233 consensus_queue.cc:260] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:08.169873 16233 ts_tablet_manager.cc:1398] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:08.170125 16233 tablet_bootstrap.cc:492] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:08.172586 16240 tablet_bootstrap.cc:654] T 8ab3748794114768bba90e903384819c P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.175130 16233 tablet_bootstrap.cc:654] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.178862 16236 tablet_bootstrap.cc:492] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:08.179145 16236 ts_tablet_manager.cc:1367] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.137s user 0.039s sys 0.054s
I20240502 08:18:08.180186 16236 raft_consensus.cc:348] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:08.180603 16236 raft_consensus.cc:374] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.181198 16236 raft_consensus.cc:729] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.181581 16236 consensus_queue.cc:260] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:08.182911 16236 ts_tablet_manager.cc:1398] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:08.186908 16233 tablet_bootstrap.cc:492] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:08.187105 16233 ts_tablet_manager.cc:1367] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240502 08:18:08.187754 16233 raft_consensus.cc:348] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.187917 16233 raft_consensus.cc:374] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.188033 16233 raft_consensus.cc:729] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.188231 16233 consensus_queue.cc:260] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.188778 16233 ts_tablet_manager.cc:1398] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:08.191948 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab3748794114768bba90e903384819c" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
W20240502 08:18:08.192546 15419 leader_election.cc:341] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:08.193548 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9468dba95134fb4be53029dc313de7d" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:08.193815 12859 raft_consensus.cc:2454] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:08.194702 15819 leader_election.cc:302] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters: db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:08.195037 16221 raft_consensus.cc:2781] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:08.195153 16221 raft_consensus.cc:482] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:08.195219 16221 raft_consensus.cc:3037] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.196597 16240 tablet_bootstrap.cc:492] T 8ab3748794114768bba90e903384819c P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:08.196741 16240 ts_tablet_manager.cc:1367] T 8ab3748794114768bba90e903384819c P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.019s
I20240502 08:18:08.197292 16240 raft_consensus.cc:348] T 8ab3748794114768bba90e903384819c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:08.197451 16240 raft_consensus.cc:374] T 8ab3748794114768bba90e903384819c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.197525 16240 raft_consensus.cc:729] T 8ab3748794114768bba90e903384819c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.197734 16240 consensus_queue.cc:260] T 8ab3748794114768bba90e903384819c P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:08.197845 16221 raft_consensus.cc:504] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:08.198233 16240 ts_tablet_manager.cc:1398] T 8ab3748794114768bba90e903384819c P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:18:08.198554 16221 leader_election.cc:288] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:08.198940 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9468dba95134fb4be53029dc313de7d" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:08.199127 15488 raft_consensus.cc:3037] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.201198 15488 raft_consensus.cc:2454] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:08.201663 15819 leader_election.cc:302] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:08.201910 16221 raft_consensus.cc:2781] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:08.202081 16221 raft_consensus.cc:686] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:08.202165 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9468dba95134fb4be53029dc313de7d" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:08.202333 12859 raft_consensus.cc:3037] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.202396 16221 consensus_queue.cc:237] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:08.204828 12859 raft_consensus.cc:2454] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:08.205329 12584 catalog_manager.cc:5484] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:08.305943 15955 raft_consensus.cc:482] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.306150 15955 raft_consensus.cc:504] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.306830 15955 leader_election.cc:288] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:08.307291 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c58721561064e97909a0caddc47a1f0" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:08.307296 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c58721561064e97909a0caddc47a1f0" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:08.307572 15488 raft_consensus.cc:2454] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:08.307557 15887 raft_consensus.cc:2454] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:08.308060 15554 leader_election.cc:302] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:08.308384 15955 raft_consensus.cc:2781] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:08.308495 15955 raft_consensus.cc:482] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:08.308610 15955 raft_consensus.cc:3037] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.310660 15955 raft_consensus.cc:504] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.311694 15955 leader_election.cc:288] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:08.311709 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c58721561064e97909a0caddc47a1f0" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:08.311744 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c58721561064e97909a0caddc47a1f0" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:08.311898 15488 raft_consensus.cc:3037] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.311925 15887 raft_consensus.cc:3037] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.314064 15488 raft_consensus.cc:2454] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:08.314107 15887 raft_consensus.cc:2454] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:08.314544 15555 leader_election.cc:302] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:08.314790 15955 raft_consensus.cc:2781] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:08.315176 15955 raft_consensus.cc:686] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:08.315464 15955 consensus_queue.cc:237] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.318157 12584 catalog_manager.cc:5484] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:08.320219 16241 raft_consensus.cc:482] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.320376 16241 raft_consensus.cc:504] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:08.321162 16241 leader_election.cc:288] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:08.322109 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a542eb208b254055b02765aceba5b405" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:08.322329 15755 raft_consensus.cc:2454] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:08.322822 12663 leader_election.cc:302] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:08.323102 16241 raft_consensus.cc:2781] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:08.323225 16241 raft_consensus.cc:482] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:08.323189 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:08.323313 16241 raft_consensus.cc:3037] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.325664 16241 raft_consensus.cc:504] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:08.326215 16241 leader_election.cc:288] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:08.327257 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a542eb208b254055b02765aceba5b405" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:08.327497 15755 raft_consensus.cc:3037] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.328303 15488 raft_consensus.cc:1261] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:08.328816 16024 consensus_queue.cc:1035] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.330183 15755 raft_consensus.cc:2454] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:08.330909 12859 raft_consensus.cc:1261] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:08.331362 12663 leader_election.cc:302] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:08.331502 16024 consensus_queue.cc:1035] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.331799 16241 raft_consensus.cc:2781] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:08.332865 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet e9468dba95134fb4be53029dc313de7d (table rebalancer_dynamic_table [id=dee6b2bb26f2495bacf2c94f8ca92ad4]). Expected version 1 got 0
I20240502 08:18:08.334120 16241 raft_consensus.cc:686] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:18:08.334636 16241 consensus_queue.cc:237] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:08.335342 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 08:18:08.335558 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 08:18:08.339428 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet fdc2defcad4c45efa0937e108ce6ad6c
I20240502 08:18:08.340276 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet fdc2defcad4c45efa0937e108ce6ad6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:08.340339 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet fdc2defcad4c45efa0937e108ce6ad6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:08.340339 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 25e96ee3f53248b6b461938fb240fc4e
I20240502 08:18:08.340559 15996 tablet_replica.cc:330] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:08.340601 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet fdc2defcad4c45efa0937e108ce6ad6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:08.340775 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 4047e22c67b8475ca3185e491e37ea98
I20240502 08:18:08.340775 15996 raft_consensus.cc:2229] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:08.340857 15993 tablet_replica.cc:330] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:08.341020 15993 raft_consensus.cc:2229] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.341091 15996 raft_consensus.cc:2258] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.341265 15993 raft_consensus.cc:2258] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.341826 15996 ts_tablet_manager.cc:1875] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.341848 15993 ts_tablet_manager.cc:1875] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.345836 16230 tablet_replica.cc:330] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:08.346046 16230 raft_consensus.cc:2229] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.346297 16230 raft_consensus.cc:2258] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.347020 16230 ts_tablet_manager.cc:1875] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.347213 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e96ee3f53248b6b461938fb240fc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:08.347494 12972 tablet_service.cc:1514] Processing DeleteTablet for tablet 4047e22c67b8475ca3185e491e37ea98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:08.347599 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e96ee3f53248b6b461938fb240fc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:08.347683 16131 tablet_replica.cc:330] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:08.347887 16131 raft_consensus.cc:2229] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:08.348183 16131 raft_consensus.cc:2258] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.348868 16131 ts_tablet_manager.cc:1875] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.349287 12840 tablet_service.cc:1514] Processing DeleteTablet for tablet 4047e22c67b8475ca3185e491e37ea98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:08.349532 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e96ee3f53248b6b461938fb240fc4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:08.349758 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 4047e22c67b8475ca3185e491e37ea98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:08.353969 16230 ts_tablet_manager.cc:1888] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.354235 15996 ts_tablet_manager.cc:1888] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.354351 15996 log.cc:1196] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/fdc2defcad4c45efa0937e108ce6ad6c
I20240502 08:18:08.354790 15996 ts_tablet_manager.cc:1909] T fdc2defcad4c45efa0937e108ce6ad6c P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:08.356329 16131 ts_tablet_manager.cc:1888] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.356707 16131 log.cc:1196] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/25e96ee3f53248b6b461938fb240fc4e
I20240502 08:18:08.356762 15993 ts_tablet_manager.cc:1888] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.356945 15993 log.cc:1196] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/fdc2defcad4c45efa0937e108ce6ad6c
I20240502 08:18:08.357183 16131 ts_tablet_manager.cc:1909] T 25e96ee3f53248b6b461938fb240fc4e P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:08.357445 15993 ts_tablet_manager.cc:1909] T fdc2defcad4c45efa0937e108ce6ad6c P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:08.357627 16230 log.cc:1196] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/fdc2defcad4c45efa0937e108ce6ad6c
I20240502 08:18:08.358109 16131 tablet_replica.cc:330] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:08.358182 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 25e96ee3f53248b6b461938fb240fc4e (table rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]) successfully deleted
I20240502 08:18:08.358294 16131 raft_consensus.cc:2229] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:08.358616 16131 raft_consensus.cc:2258] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.359000 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet fdc2defcad4c45efa0937e108ce6ad6c (table rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]) successfully deleted
I20240502 08:18:08.359266 16131 ts_tablet_manager.cc:1875] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.359575 15996 tablet_replica.cc:330] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:08.359781 15996 raft_consensus.cc:2229] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.359997 15996 raft_consensus.cc:2258] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.360037 16230 ts_tablet_manager.cc:1909] T fdc2defcad4c45efa0937e108ce6ad6c P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:08.361416 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet fdc2defcad4c45efa0937e108ce6ad6c (table rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]) successfully deleted
I20240502 08:18:08.361948 15993 tablet_replica.cc:330] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:08.362082 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet fdc2defcad4c45efa0937e108ce6ad6c (table rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]) successfully deleted
I20240502 08:18:08.362529 15993 raft_consensus.cc:2229] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.362445 16230 tablet_replica.cc:330] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:08.362843 16230 raft_consensus.cc:2229] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.363341 16230 raft_consensus.cc:2258] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.363484 15993 raft_consensus.cc:2258] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.364028 15996 ts_tablet_manager.cc:1875] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.364653 15993 ts_tablet_manager.cc:1875] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.364627 12584 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet e9468dba95134fb4be53029dc313de7d (table rebalancer_dynamic_table [id=dee6b2bb26f2495bacf2c94f8ca92ad4]). Expected version 1 got 0
I20240502 08:18:08.365010 16230 ts_tablet_manager.cc:1875] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.369515 16273 tablet.cc:1722] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:08.371855 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet e9468dba95134fb4be53029dc313de7d (table rebalancer_dynamic_table [id=dee6b2bb26f2495bacf2c94f8ca92ad4]). Expected version 1 got 0
W20240502 08:18:08.372728 13168 tablet.cc:2369] T 8ab3748794114768bba90e903384819c P 6a0b05f3250d415590b0edf6b44fbe1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:08.372416 16270 tablet.cc:1722] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:08.375911 16273 tablet.cc:1620] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:08.375921 16270 tablet.cc:1620] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:08.378486 12584 catalog_manager.cc:5484] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:08.380075 16131 ts_tablet_manager.cc:1888] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.381815 15996 ts_tablet_manager.cc:1888] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.383266 12579 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet e9468dba95134fb4be53029dc313de7d (table rebalancer_dynamic_table [id=dee6b2bb26f2495bacf2c94f8ca92ad4]). Expected version 1 got 0
I20240502 08:18:08.383060 16131 log.cc:1196] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/4047e22c67b8475ca3185e491e37ea98
I20240502 08:18:08.385581 16230 ts_tablet_manager.cc:1888] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.385715 16230 log.cc:1196] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/25e96ee3f53248b6b461938fb240fc4e
I20240502 08:18:08.386189 16131 ts_tablet_manager.cc:1909] T 4047e22c67b8475ca3185e491e37ea98 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:08.387087 12586 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=dee6b2bb26f2495bacf2c94f8ca92ad4] alter complete (version 1)
I20240502 08:18:08.388129 15996 log.cc:1196] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/25e96ee3f53248b6b461938fb240fc4e
I20240502 08:18:08.388517 15993 ts_tablet_manager.cc:1888] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.388577 15996 ts_tablet_manager.cc:1909] T 25e96ee3f53248b6b461938fb240fc4e P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:08.388796 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 4047e22c67b8475ca3185e491e37ea98 (table rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]) successfully deleted
I20240502 08:18:08.380339 16275 tablet.cc:1722] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:08.389802 15993 log.cc:1196] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/4047e22c67b8475ca3185e491e37ea98
I20240502 08:18:08.389976 16275 tablet.cc:1620] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:08.390610 15993 ts_tablet_manager.cc:1909] T 4047e22c67b8475ca3185e491e37ea98 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:08.390609 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a542eb208b254055b02765aceba5b405" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:08.390827 15887 raft_consensus.cc:2454] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:08.391139 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a542eb208b254055b02765aceba5b405" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:08.391314 15886 raft_consensus.cc:3037] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.391327 16230 ts_tablet_manager.cc:1909] T 25e96ee3f53248b6b461938fb240fc4e P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:08.391379 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 25e96ee3f53248b6b461938fb240fc4e (table rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]) successfully deleted
I20240502 08:18:08.391672 15996 tablet_replica.cc:330] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:08.391853 15996 raft_consensus.cc:2229] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.392074 15996 raft_consensus.cc:2258] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.392700 15996 ts_tablet_manager.cc:1875] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.392901 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 4047e22c67b8475ca3185e491e37ea98 (table rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]) successfully deleted
I20240502 08:18:08.393152 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 25e96ee3f53248b6b461938fb240fc4e (table rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]) successfully deleted
I20240502 08:18:08.393253 15886 raft_consensus.cc:2454] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:08.396993 15996 ts_tablet_manager.cc:1888] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.397104 15996 log.cc:1196] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/4047e22c67b8475ca3185e491e37ea98
I20240502 08:18:08.397456 15996 ts_tablet_manager.cc:1909] T 4047e22c67b8475ca3185e491e37ea98 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:08.398196 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 4047e22c67b8475ca3185e491e37ea98 (table rebalance_test_table_16 [id=1664caf28b514be0b0762041ae994a93]) successfully deleted
I20240502 08:18:08.498509 12586 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:08.504951 16270 tablet.cc:1722] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:08.505427 16270 tablet.cc:1620] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:08.505551 16273 tablet.cc:1722] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:08.505980 16273 tablet.cc:1620] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:08.506738 16275 tablet.cc:1722] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:08.507174 16275 tablet.cc:1620] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:08.512439 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=dee6b2bb26f2495bacf2c94f8ca92ad4] alter complete (version 2)
I20240502 08:18:08.539093 12859 raft_consensus.cc:3037] T 8ab3748794114768bba90e903384819c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.541571 16242 consensus_queue.cc:1035] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.546267 16242 consensus_queue.cc:1035] T 8ab3748794114768bba90e903384819c P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.616639 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:08.624315 16270 tablet.cc:1722] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:08.626370 16273 tablet.cc:1722] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:08.627203 16275 tablet.cc:1722] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:08.634434 12586 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=dee6b2bb26f2495bacf2c94f8ca92ad4] alter complete (version 3)
I20240502 08:18:08.737579 12586 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:08.737802 12586 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:08.740754 12586 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet e9468dba95134fb4be53029dc313de7d
I20240502 08:18:08.741470 12840 tablet_service.cc:1514] Processing DeleteTablet for tablet e9468dba95134fb4be53029dc313de7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:08.741706 15996 tablet_replica.cc:330] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:08.741703 15866 tablet_service.cc:1514] Processing DeleteTablet for tablet e9468dba95134fb4be53029dc313de7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:08.741891 15996 raft_consensus.cc:2229] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.742058 15996 raft_consensus.cc:2258] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.742868 15466 tablet_service.cc:1514] Processing DeleteTablet for tablet e9468dba95134fb4be53029dc313de7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:08.743069 15996 ts_tablet_manager.cc:1875] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.743204 16281 tablet_replica.cc:330] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:08.743340 16281 raft_consensus.cc:2229] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:08.743593 16281 raft_consensus.cc:2258] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.744388 16281 ts_tablet_manager.cc:1875] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.745200 16282 tablet_replica.cc:330] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:08.745438 16282 raft_consensus.cc:2229] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.745649 16282 raft_consensus.cc:2258] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.747969 16282 ts_tablet_manager.cc:1875] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.750157 15996 ts_tablet_manager.cc:1888] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:08.750303 15996 log.cc:1196] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/e9468dba95134fb4be53029dc313de7d
I20240502 08:18:08.750758 15996 ts_tablet_manager.cc:1909] T e9468dba95134fb4be53029dc313de7d P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:08.751633 16281 ts_tablet_manager.cc:1888] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:08.751670 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet e9468dba95134fb4be53029dc313de7d (table rebalancer_dynamic_table_new_name [id=dee6b2bb26f2495bacf2c94f8ca92ad4]) successfully deleted
I20240502 08:18:08.751828 16281 log.cc:1196] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/e9468dba95134fb4be53029dc313de7d
I20240502 08:18:08.752280 16281 ts_tablet_manager.cc:1909] T e9468dba95134fb4be53029dc313de7d P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:08.752875 16241 consensus_queue.cc:1035] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.753288 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet e9468dba95134fb4be53029dc313de7d (table rebalancer_dynamic_table_new_name [id=dee6b2bb26f2495bacf2c94f8ca92ad4]) successfully deleted
I20240502 08:18:08.754480 16282 ts_tablet_manager.cc:1888] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:08.754604 16282 log.cc:1196] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/e9468dba95134fb4be53029dc313de7d
I20240502 08:18:08.755050 16282 ts_tablet_manager.cc:1909] T e9468dba95134fb4be53029dc313de7d P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:08.755872 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet e9468dba95134fb4be53029dc313de7d (table rebalancer_dynamic_table_new_name [id=dee6b2bb26f2495bacf2c94f8ca92ad4]) successfully deleted
I20240502 08:18:08.757485 16187 consensus_queue.cc:1035] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.844056 12586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:08.850314 12586 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 08:18:08.850659 12586 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 08:18:08.851198 15466 tablet_service.cc:1467] Processing CreateTablet for tablet 2800e64ca3f643449de1214ea0e83b31 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3788dd2b268847f9a64083ddf308ed81]), partition=
I20240502 08:18:08.851506 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 2800e64ca3f643449de1214ea0e83b31 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3788dd2b268847f9a64083ddf308ed81]), partition=
I20240502 08:18:08.851620 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2800e64ca3f643449de1214ea0e83b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.851781 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2800e64ca3f643449de1214ea0e83b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.854171 12586 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 1319f55da0b348ac8a325e7d04f0a11d
I20240502 08:18:08.854700 12586 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet ed9fd50fd4474d25883861f03d6333c0
I20240502 08:18:08.854990 12840 tablet_service.cc:1514] Processing DeleteTablet for tablet 1319f55da0b348ac8a325e7d04f0a11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:08.855111 12972 tablet_service.cc:1514] Processing DeleteTablet for tablet 1319f55da0b348ac8a325e7d04f0a11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:08.855254 15996 tablet_replica.cc:330] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:08.855377 16131 tablet_replica.cc:330] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:08.855388 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9fd50fd4474d25883861f03d6333c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:08.855443 15996 raft_consensus.cc:2229] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.855573 16131 raft_consensus.cc:2229] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.855674 15996 raft_consensus.cc:2258] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.855784 16131 raft_consensus.cc:2258] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.855931 12586 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 385630fa0b3d4fa7b48af2aa7852d2d5
I20240502 08:18:08.856352 15996 ts_tablet_manager.cc:1875] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.856371 16131 ts_tablet_manager.cc:1875] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.857251 16287 tablet_bootstrap.cc:492] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:08.858114 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 2800e64ca3f643449de1214ea0e83b31 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3788dd2b268847f9a64083ddf308ed81]), partition=
I20240502 08:18:08.858451 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2800e64ca3f643449de1214ea0e83b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.858690 16287 tablet_bootstrap.cc:654] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.858901 12710 tablet_service.cc:1514] Processing DeleteTablet for tablet 385630fa0b3d4fa7b48af2aa7852d2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:08.859342 16230 tablet_replica.cc:330] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:08.859542 16230 raft_consensus.cc:2229] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.859741 16230 raft_consensus.cc:2258] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.860423 16230 ts_tablet_manager.cc:1875] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.860962 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 385630fa0b3d4fa7b48af2aa7852d2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:08.862591 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 385630fa0b3d4fa7b48af2aa7852d2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:08.862632 16131 ts_tablet_manager.cc:1888] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.862754 16131 log.cc:1196] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/1319f55da0b348ac8a325e7d04f0a11d
I20240502 08:18:08.858903 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet 1319f55da0b348ac8a325e7d04f0a11d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:08.863008 12840 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9fd50fd4474d25883861f03d6333c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:08.863235 16131 ts_tablet_manager.cc:1909] T 1319f55da0b348ac8a325e7d04f0a11d P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:08.864248 16131 tablet_replica.cc:330] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:08.864289 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 1319f55da0b348ac8a325e7d04f0a11d (table rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]) successfully deleted
I20240502 08:18:08.864419 16131 raft_consensus.cc:2229] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:08.864427 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet ed9fd50fd4474d25883861f03d6333c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:08 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:08.864723 15993 tablet_replica.cc:330] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:08.864755 16131 raft_consensus.cc:2258] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.864930 15993 raft_consensus.cc:2229] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.865181 15993 raft_consensus.cc:2258] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.865373 16131 ts_tablet_manager.cc:1875] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.865859 15993 ts_tablet_manager.cc:1875] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.867991 15955 consensus_queue.cc:1035] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.871644 16131 ts_tablet_manager.cc:1888] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.871769 16131 log.cc:1196] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/ed9fd50fd4474d25883861f03d6333c0
I20240502 08:18:08.872150 16131 ts_tablet_manager.cc:1909] T ed9fd50fd4474d25883861f03d6333c0 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:08.873682 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet ed9fd50fd4474d25883861f03d6333c0 (table rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]) successfully deleted
I20240502 08:18:08.873811 16131 tablet_replica.cc:330] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:08.874015 16131 raft_consensus.cc:2229] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.874231 16131 raft_consensus.cc:2258] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.874867 16131 ts_tablet_manager.cc:1875] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.877704 15955 consensus_queue.cc:1035] T 3c58721561064e97909a0caddc47a1f0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.878612 16287 tablet_bootstrap.cc:492] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:08.878753 16287 ts_tablet_manager.cc:1367] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.007s
I20240502 08:18:08.879541 16287 raft_consensus.cc:348] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.879772 16287 raft_consensus.cc:374] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.879850 16287 raft_consensus.cc:729] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.880069 16287 consensus_queue.cc:260] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.880587 16287 ts_tablet_manager.cc:1398] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:08.880602 16131 ts_tablet_manager.cc:1888] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.880731 16131 log.cc:1196] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/385630fa0b3d4fa7b48af2aa7852d2d5
I20240502 08:18:08.881121 16131 ts_tablet_manager.cc:1909] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:08.882086 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 385630fa0b3d4fa7b48af2aa7852d2d5 (table rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]) successfully deleted
I20240502 08:18:08.885357 16292 tablet_bootstrap.cc:492] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:08.886837 16292 tablet_bootstrap.cc:654] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.894977 16292 tablet_bootstrap.cc:492] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:08.895002 16290 tablet_bootstrap.cc:492] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:08.895176 16292 ts_tablet_manager.cc:1367] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20240502 08:18:08.896003 16292 raft_consensus.cc:348] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.896262 16292 raft_consensus.cc:374] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.896397 16292 raft_consensus.cc:729] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.896448 15996 ts_tablet_manager.cc:1888] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.896567 15996 log.cc:1196] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/1319f55da0b348ac8a325e7d04f0a11d
I20240502 08:18:08.896765 16292 consensus_queue.cc:260] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.897018 15996 ts_tablet_manager.cc:1909] T 1319f55da0b348ac8a325e7d04f0a11d P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:08.897966 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 1319f55da0b348ac8a325e7d04f0a11d (table rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]) successfully deleted
I20240502 08:18:08.899801 16292 ts_tablet_manager.cc:1398] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 08:18:08.901209 16290 tablet_bootstrap.cc:654] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.901667 15993 ts_tablet_manager.cc:1888] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.901793 15993 log.cc:1196] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/ed9fd50fd4474d25883861f03d6333c0
I20240502 08:18:08.902223 15993 ts_tablet_manager.cc:1909] T ed9fd50fd4474d25883861f03d6333c0 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:08.902411 16230 ts_tablet_manager.cc:1888] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.902619 16230 log.cc:1196] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/385630fa0b3d4fa7b48af2aa7852d2d5
I20240502 08:18:08.902992 16230 ts_tablet_manager.cc:1909] T 385630fa0b3d4fa7b48af2aa7852d2d5 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:08.903512 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet ed9fd50fd4474d25883861f03d6333c0 (table rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]) successfully deleted
I20240502 08:18:08.903880 16230 tablet_replica.cc:330] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:08.903960 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 385630fa0b3d4fa7b48af2aa7852d2d5 (table rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]) successfully deleted
I20240502 08:18:08.904072 16230 raft_consensus.cc:2229] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:08.904371 16230 raft_consensus.cc:2258] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.905048 16230 ts_tablet_manager.cc:1875] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.906663 15996 tablet_replica.cc:330] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:08.906911 15996 raft_consensus.cc:2229] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:08.907277 15996 raft_consensus.cc:2258] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.908007 15996 ts_tablet_manager.cc:1875] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.908082 16228 raft_consensus.cc:482] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.908253 16228 raft_consensus.cc:504] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.908918 16228 leader_election.cc:288] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:08.909511 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2800e64ca3f643449de1214ea0e83b31" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:08.909756 15488 raft_consensus.cc:2454] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:08.910187 16230 ts_tablet_manager.cc:1888] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.910367 16230 log.cc:1196] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/1319f55da0b348ac8a325e7d04f0a11d
I20240502 08:18:08.910645 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2800e64ca3f643449de1214ea0e83b31" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:08.910811 16230 ts_tablet_manager.cc:1909] T 1319f55da0b348ac8a325e7d04f0a11d P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:08.911397 15688 leader_election.cc:302] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:08.911784 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 1319f55da0b348ac8a325e7d04f0a11d (table rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]) successfully deleted
W20240502 08:18:08.911893 15690 leader_election.cc:341] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:08.912281 16228 raft_consensus.cc:2781] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:08.912369 16228 raft_consensus.cc:482] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:08.912460 16228 raft_consensus.cc:3037] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.913100 16290 tablet_bootstrap.cc:492] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:08.913216 16290 ts_tablet_manager.cc:1367] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.000s
I20240502 08:18:08.913946 16290 raft_consensus.cc:348] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.914119 16290 raft_consensus.cc:374] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.914187 16290 raft_consensus.cc:729] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.914394 16290 consensus_queue.cc:260] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.914446 16228 raft_consensus.cc:504] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.914925 16290 ts_tablet_manager.cc:1398] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:08.915489 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2800e64ca3f643449de1214ea0e83b31" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:08.915530 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2800e64ca3f643449de1214ea0e83b31" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:08.915693 15622 raft_consensus.cc:3037] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.915712 15488 raft_consensus.cc:3037] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.916697 15996 ts_tablet_manager.cc:1888] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.916826 15996 log.cc:1196] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/385630fa0b3d4fa7b48af2aa7852d2d5
I20240502 08:18:08.917160 15996 ts_tablet_manager.cc:1909] T 385630fa0b3d4fa7b48af2aa7852d2d5 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:08.917937 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 385630fa0b3d4fa7b48af2aa7852d2d5 (table rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]) successfully deleted
I20240502 08:18:08.918124 15996 tablet_replica.cc:330] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:08.918287 15996 raft_consensus.cc:2229] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:08.918258 15488 raft_consensus.cc:2454] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:08.918323 15622 raft_consensus.cc:2454] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:08.918519 15996 raft_consensus.cc:2258] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:08.919054 15688 leader_election.cc:302] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:08.919343 15996 ts_tablet_manager.cc:1875] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:08.919626 16297 raft_consensus.cc:2781] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:08.915292 16228 leader_election.cc:288] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:08.921198 16297 raft_consensus.cc:686] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:08.921497 16297 consensus_queue.cc:237] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.924150 12584 catalog_manager.cc:5484] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:08.925633 15996 ts_tablet_manager.cc:1888] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 08:18:08.925762 15996 log.cc:1196] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/ed9fd50fd4474d25883861f03d6333c0
I20240502 08:18:08.926149 15996 ts_tablet_manager.cc:1909] T ed9fd50fd4474d25883861f03d6333c0 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:08.927023 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet ed9fd50fd4474d25883861f03d6333c0 (table rebalance_test_table_18 [id=6cba609dfa7a42e991ebe4b93bdde0a8]) successfully deleted
I20240502 08:18:08.927528 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:08.934805 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 18c58e2a9a814bcb8ff6bb9e3011084f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f19b8479d1b74f17b2ee9e2117749102]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.935254 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c58e2a9a814bcb8ff6bb9e3011084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.935344 15866 tablet_service.cc:1467] Processing CreateTablet for tablet 18c58e2a9a814bcb8ff6bb9e3011084f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f19b8479d1b74f17b2ee9e2117749102]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.935812 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c58e2a9a814bcb8ff6bb9e3011084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.936067 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 18c58e2a9a814bcb8ff6bb9e3011084f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f19b8479d1b74f17b2ee9e2117749102]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.936421 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c58e2a9a814bcb8ff6bb9e3011084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.939061 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 646aec80c3634b6ba8493832ff4294a0 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f19b8479d1b74f17b2ee9e2117749102]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.939069 15466 tablet_service.cc:1467] Processing CreateTablet for tablet 646aec80c3634b6ba8493832ff4294a0 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f19b8479d1b74f17b2ee9e2117749102]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.939430 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646aec80c3634b6ba8493832ff4294a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.939462 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646aec80c3634b6ba8493832ff4294a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.940075 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 646aec80c3634b6ba8493832ff4294a0 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f19b8479d1b74f17b2ee9e2117749102]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.940479 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646aec80c3634b6ba8493832ff4294a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.943631 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7190137f2949de8df2cdf447802db1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f19b8479d1b74f17b2ee9e2117749102]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.943979 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7190137f2949de8df2cdf447802db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.946846 16287 tablet_bootstrap.cc:492] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:08.946962 16292 tablet_bootstrap.cc:492] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:08.948274 15600 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7190137f2949de8df2cdf447802db1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f19b8479d1b74f17b2ee9e2117749102]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.948343 16287 tablet_bootstrap.cc:654] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.948410 16292 tablet_bootstrap.cc:654] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.948592 15600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7190137f2949de8df2cdf447802db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.949553 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7190137f2949de8df2cdf447802db1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f19b8479d1b74f17b2ee9e2117749102]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.949949 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7190137f2949de8df2cdf447802db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.954046 16300 tablet_bootstrap.cc:492] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:08.954275 16290 tablet_bootstrap.cc:492] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:08.955667 16300 tablet_bootstrap.cc:654] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.957443 16290 tablet_bootstrap.cc:654] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.962370 16292 tablet_bootstrap.cc:492] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:08.962592 16292 ts_tablet_manager.cc:1367] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20240502 08:18:08.963436 16292 raft_consensus.cc:348] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.963824 16292 raft_consensus.cc:374] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.963938 16292 raft_consensus.cc:729] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.964251 16292 consensus_queue.cc:260] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.964375 16300 tablet_bootstrap.cc:492] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:08.964504 16300 ts_tablet_manager.cc:1367] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240502 08:18:08.964854 16292 ts_tablet_manager.cc:1398] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:08.965183 16292 tablet_bootstrap.cc:492] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:08.965394 16300 raft_consensus.cc:348] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.965634 16300 raft_consensus.cc:374] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.965819 16300 raft_consensus.cc:729] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.966315 16300 consensus_queue.cc:260] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.968750 16300 ts_tablet_manager.cc:1398] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:08.969069 16300 tablet_bootstrap.cc:492] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:08.969786 16292 tablet_bootstrap.cc:654] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.970299 16287 tablet_bootstrap.cc:492] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:08.970491 16287 ts_tablet_manager.cc:1367] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.005s
I20240502 08:18:08.971210 16287 raft_consensus.cc:348] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.971437 16287 raft_consensus.cc:374] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.971511 16287 raft_consensus.cc:729] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.971765 16287 consensus_queue.cc:260] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.972573 16290 tablet_bootstrap.cc:492] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:08.972795 16290 ts_tablet_manager.cc:1367] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.004s
I20240502 08:18:08.973117 16300 tablet_bootstrap.cc:654] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.974236 16290 raft_consensus.cc:348] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.974421 16292 tablet_bootstrap.cc:492] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:08.974539 16290 raft_consensus.cc:374] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.974568 16292 ts_tablet_manager.cc:1367] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240502 08:18:08.974664 16290 raft_consensus.cc:729] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.974795 16287 ts_tablet_manager.cc:1398] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 08:18:08.975006 16290 consensus_queue.cc:260] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.975214 16292 raft_consensus.cc:348] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.975385 16292 raft_consensus.cc:374] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.975463 16292 raft_consensus.cc:729] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.975705 16292 consensus_queue.cc:260] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.975853 16290 ts_tablet_manager.cc:1398] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:08.976255 16292 ts_tablet_manager.cc:1398] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:08.976574 16292 tablet_bootstrap.cc:492] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:08.977362 16290 tablet_bootstrap.cc:492] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:08.977396 16301 tablet_bootstrap.cc:492] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:08.978412 16292 tablet_bootstrap.cc:654] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.980415 16290 tablet_bootstrap.cc:654] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.982947 16300 tablet_bootstrap.cc:492] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:08.983192 16300 ts_tablet_manager.cc:1367] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240502 08:18:08.983990 16301 tablet_bootstrap.cc:654] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.983916 16300 raft_consensus.cc:348] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.984194 16300 raft_consensus.cc:374] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.984325 16300 raft_consensus.cc:729] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.984624 16300 consensus_queue.cc:260] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.987534 16300 ts_tablet_manager.cc:1398] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:08.987670 16292 tablet_bootstrap.cc:492] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:08.987885 16292 ts_tablet_manager.cc:1367] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240502 08:18:08.988620 16292 raft_consensus.cc:348] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.988888 16292 raft_consensus.cc:374] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.988994 16292 raft_consensus.cc:729] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.989292 16292 consensus_queue.cc:260] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.991148 16292 ts_tablet_manager.cc:1398] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:08.991817 16290 tablet_bootstrap.cc:492] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:08.991963 16290 ts_tablet_manager.cc:1367] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240502 08:18:08.992527 16290 raft_consensus.cc:348] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.992729 16290 raft_consensus.cc:374] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.992836 16290 raft_consensus.cc:729] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.993019 16290 consensus_queue.cc:260] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:08.993157 16301 tablet_bootstrap.cc:492] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:08.993372 16301 ts_tablet_manager.cc:1367] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.004s
I20240502 08:18:08.993422 16290 ts_tablet_manager.cc:1398] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:18:08.994103 16301 raft_consensus.cc:348] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.994287 16301 raft_consensus.cc:374] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.994377 16301 raft_consensus.cc:729] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.994604 16301 consensus_queue.cc:260] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:08.995234 16301 ts_tablet_manager.cc:1398] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:09.020536 16277 raft_consensus.cc:482] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.020702 16277 raft_consensus.cc:504] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:09.021589 16277 leader_election.cc:288] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:09.032335 12586 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:09.039847 15622 raft_consensus.cc:1261] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:09.040481 16297 consensus_queue.cc:1035] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.047928 15488 raft_consensus.cc:1261] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:09.048534 16297 consensus_queue.cc:1035] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.054492 12586 catalog_manager.cc:5576] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983) does not have the latest schema for tablet 2800e64ca3f643449de1214ea0e83b31 (table rebalancer_dynamic_table [id=3788dd2b268847f9a64083ddf308ed81]). Expected version 1 got 0
I20240502 08:18:09.057502 16317 tablet.cc:1722] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:09.059044 12584 catalog_manager.cc:5576] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983) does not have the latest schema for tablet 2800e64ca3f643449de1214ea0e83b31 (table rebalancer_dynamic_table [id=3788dd2b268847f9a64083ddf308ed81]). Expected version 1 got 0
I20240502 08:18:09.061982 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646aec80c3634b6ba8493832ff4294a0" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:09.062232 12730 raft_consensus.cc:2454] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:09.062808 15419 leader_election.cc:302] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:09.063112 16317 tablet.cc:1620] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:09.062889 16273 tablet.cc:1722] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:09.063714 16273 tablet.cc:1620] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:09.064137 16277 raft_consensus.cc:2781] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.064249 16277 raft_consensus.cc:482] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.064335 16277 raft_consensus.cc:3037] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.065500 16316 tablet.cc:1722] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:09.066236 16316 tablet.cc:1620] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:09.066506 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet 2800e64ca3f643449de1214ea0e83b31 (table rebalancer_dynamic_table [id=3788dd2b268847f9a64083ddf308ed81]). Expected version 1 got 0
I20240502 08:18:09.066557 16221 raft_consensus.cc:482] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.066730 16221 raft_consensus.cc:504] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:09.067391 16221 leader_election.cc:288] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:09.067981 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7190137f2949de8df2cdf447802db1" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:09.068179 15621 raft_consensus.cc:2454] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:09.068841 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7190137f2949de8df2cdf447802db1" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:09.069063 15755 raft_consensus.cc:2454] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:09.069593 15822 leader_election.cc:302] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:09.070289 16221 raft_consensus.cc:2781] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.070391 16221 raft_consensus.cc:482] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.070506 16221 raft_consensus.cc:3037] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.070729 16277 raft_consensus.cc:504] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:09.071369 16277 leader_election.cc:288] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:09.071920 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=3788dd2b268847f9a64083ddf308ed81] alter complete (version 1)
I20240502 08:18:09.073283 16221 raft_consensus.cc:504] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:09.073940 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646aec80c3634b6ba8493832ff4294a0" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:09.074151 12730 raft_consensus.cc:3037] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.074373 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7190137f2949de8df2cdf447802db1" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:09.074601 15755 raft_consensus.cc:3037] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.076674 12730 raft_consensus.cc:2454] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:09.078299 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646aec80c3634b6ba8493832ff4294a0" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:09.078513 15755 raft_consensus.cc:2454] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:09.078925 15818 leader_election.cc:302] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:09.079043 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646aec80c3634b6ba8493832ff4294a0" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:09.079198 16024 raft_consensus.cc:2781] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.078513 15754 raft_consensus.cc:2454] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:09.081164 15419 leader_election.cc:302] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters: 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:09.081136 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7190137f2949de8df2cdf447802db1" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:09.081333 15621 raft_consensus.cc:3037] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.084153 15621 raft_consensus.cc:2454] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:09.084846 16277 raft_consensus.cc:2781] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.084998 16277 raft_consensus.cc:686] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:09.085300 16277 consensus_queue.cc:237] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:09.086416 16024 raft_consensus.cc:686] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:09.086745 16024 consensus_queue.cc:237] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:09.089543 16221 leader_election.cc:288] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:09.096330 12584 catalog_manager.cc:5484] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:09.097395 12586 catalog_manager.cc:5484] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:09.173606 12586 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:09.182917 16316 tablet.cc:1722] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:09.183390 16316 tablet.cc:1620] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:09.184458 16273 tablet.cc:1722] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:09.184513 16317 tablet.cc:1722] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:09.184929 16273 tablet.cc:1620] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:09.184963 16317 tablet.cc:1620] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:09.194137 12586 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=3788dd2b268847f9a64083ddf308ed81] alter complete (version 2)
I20240502 08:18:09.212817 16298 raft_consensus.cc:482] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.212982 16298 raft_consensus.cc:504] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:09.213678 16298 leader_election.cc:288] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:09.214069 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c58e2a9a814bcb8ff6bb9e3011084f" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:09.214154 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c58e2a9a814bcb8ff6bb9e3011084f" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:09.214254 15886 raft_consensus.cc:2454] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:09.214576 15621 raft_consensus.cc:2454] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:09.214708 15689 leader_election.cc:302] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:09.214958 16298 raft_consensus.cc:2781] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.215073 16298 raft_consensus.cc:482] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.215152 16298 raft_consensus.cc:3037] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.216974 16298 raft_consensus.cc:504] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:09.217451 16298 leader_election.cc:288] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:09.217849 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c58e2a9a814bcb8ff6bb9e3011084f" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:09.217839 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18c58e2a9a814bcb8ff6bb9e3011084f" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:09.218022 15886 raft_consensus.cc:3037] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.218046 15621 raft_consensus.cc:3037] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.220647 15621 raft_consensus.cc:2454] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:09.220664 15886 raft_consensus.cc:2454] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:09.221030 15690 leader_election.cc:302] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:09.221274 16298 raft_consensus.cc:2781] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.221503 16298 raft_consensus.cc:686] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:09.222110 16298 consensus_queue.cc:237] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:09.224969 12579 catalog_manager.cc:5484] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:09.297075 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:09.304545 16316 tablet.cc:1722] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:09.305032 16317 tablet.cc:1722] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:09.309060 16273 tablet.cc:1722] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:09.313165 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=3788dd2b268847f9a64083ddf308ed81] alter complete (version 3)
I20240502 08:18:09.414691 12586 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:09.414939 12586 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:09.417784 12586 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 2800e64ca3f643449de1214ea0e83b31
I20240502 08:18:09.418555 15466 tablet_service.cc:1514] Processing DeleteTablet for tablet 2800e64ca3f643449de1214ea0e83b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:09 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:09.418628 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 2800e64ca3f643449de1214ea0e83b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:09 UTC) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:09.418756 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 2800e64ca3f643449de1214ea0e83b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:09 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:09.419350 16328 tablet_replica.cc:330] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:09.419489 16326 tablet_replica.cc:330] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:09.419557 16328 raft_consensus.cc:2229] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:09.419659 16327 tablet_replica.cc:330] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:09.419682 16326 raft_consensus.cc:2229] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:09.419876 16326 raft_consensus.cc:2258] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.420104 16328 raft_consensus.cc:2258] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.420578 16326 ts_tablet_manager.cc:1875] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:09.422914 16327 raft_consensus.cc:2229] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:09.423066 16328 ts_tablet_manager.cc:1875] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:09.423187 16327 raft_consensus.cc:2258] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:09.424201 16327 ts_tablet_manager.cc:1875] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:09.425910 16326 ts_tablet_manager.cc:1888] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:09.426090 16326 log.cc:1196] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/2800e64ca3f643449de1214ea0e83b31
I20240502 08:18:09.426611 16326 ts_tablet_manager.cc:1909] T 2800e64ca3f643449de1214ea0e83b31 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:09.427726 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 2800e64ca3f643449de1214ea0e83b31 (table rebalancer_dynamic_table_new_name [id=3788dd2b268847f9a64083ddf308ed81]) successfully deleted
I20240502 08:18:09.429428 16327 ts_tablet_manager.cc:1888] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:09.429481 16328 ts_tablet_manager.cc:1888] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:09.429540 16327 log.cc:1196] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/2800e64ca3f643449de1214ea0e83b31
I20240502 08:18:09.429634 16328 log.cc:1196] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/2800e64ca3f643449de1214ea0e83b31
I20240502 08:18:09.429888 16327 ts_tablet_manager.cc:1909] T 2800e64ca3f643449de1214ea0e83b31 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting consensus metadata
I20240502 08:18:09.430016 16328 ts_tablet_manager.cc:1909] T 2800e64ca3f643449de1214ea0e83b31 P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:09.430716 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 2800e64ca3f643449de1214ea0e83b31 (table rebalancer_dynamic_table_new_name [id=3788dd2b268847f9a64083ddf308ed81]) successfully deleted
I20240502 08:18:09.430943 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 2800e64ca3f643449de1214ea0e83b31 (table rebalancer_dynamic_table_new_name [id=3788dd2b268847f9a64083ddf308ed81]) successfully deleted
I20240502 08:18:09.479738 16024 consensus_queue.cc:1035] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.483947 16024 consensus_queue.cc:1035] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.520010 12586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:09.526700 15733 tablet_service.cc:1467] Processing CreateTablet for tablet be2e95abf26f4c8284805524557079cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9ffd254d93bc4b49884083efe9c8d8f0]), partition=
I20240502 08:18:09.526952 15600 tablet_service.cc:1467] Processing CreateTablet for tablet be2e95abf26f4c8284805524557079cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9ffd254d93bc4b49884083efe9c8d8f0]), partition=
I20240502 08:18:09.527109 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2e95abf26f4c8284805524557079cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.527283 15600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2e95abf26f4c8284805524557079cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.527429 15466 tablet_service.cc:1467] Processing CreateTablet for tablet be2e95abf26f4c8284805524557079cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9ffd254d93bc4b49884083efe9c8d8f0]), partition=
I20240502 08:18:09.527817 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2e95abf26f4c8284805524557079cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.532699 16332 tablet_bootstrap.cc:492] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:09.534554 16332 tablet_bootstrap.cc:654] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.535881 16334 tablet_bootstrap.cc:492] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:09.536389 16332 tablet_bootstrap.cc:492] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:09.536691 16333 tablet_bootstrap.cc:492] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:09.536723 16332 ts_tablet_manager.cc:1367] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 08:18:09.538117 16334 tablet_bootstrap.cc:654] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.538007 16332 raft_consensus.cc:348] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:09.538157 16333 tablet_bootstrap.cc:654] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.538199 16332 raft_consensus.cc:374] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.538318 16332 raft_consensus.cc:729] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.538650 16332 consensus_queue.cc:260] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:09.539217 16332 ts_tablet_manager.cc:1398] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:09.540352 16333 tablet_bootstrap.cc:492] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:09.540479 16333 ts_tablet_manager.cc:1367] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 08:18:09.541267 16333 raft_consensus.cc:348] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:09.541498 16333 raft_consensus.cc:374] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.541616 16333 raft_consensus.cc:729] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.541823 16333 consensus_queue.cc:260] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:09.542315 16334 tablet_bootstrap.cc:492] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:09.542343 16333 ts_tablet_manager.cc:1398] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:09.542490 16334 ts_tablet_manager.cc:1367] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240502 08:18:09.543220 16334 raft_consensus.cc:348] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:09.543450 16334 raft_consensus.cc:374] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.543565 16334 raft_consensus.cc:729] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.543854 16334 consensus_queue.cc:260] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:09.544441 16334 ts_tablet_manager.cc:1398] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:09.587826 16293 raft_consensus.cc:482] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.587966 16293 raft_consensus.cc:504] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:09.588579 16293 leader_election.cc:288] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:09.588909 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2e95abf26f4c8284805524557079cd" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:09.589010 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2e95abf26f4c8284805524557079cd" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:09.589080 15755 raft_consensus.cc:2454] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:09.589227 15488 raft_consensus.cc:2454] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:09.589453 15553 leader_election.cc:302] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:09.589704 16293 raft_consensus.cc:2781] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.589800 16293 raft_consensus.cc:482] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.589876 16293 raft_consensus.cc:3037] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.591703 16293 raft_consensus.cc:504] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:09.592233 16293 leader_election.cc:288] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:09.592559 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2e95abf26f4c8284805524557079cd" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:09.592636 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2e95abf26f4c8284805524557079cd" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:09.592713 15755 raft_consensus.cc:3037] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.592830 15488 raft_consensus.cc:3037] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.594553 15755 raft_consensus.cc:2454] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:09.594604 15488 raft_consensus.cc:2454] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:09.594897 15553 leader_election.cc:302] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:09.595183 16293 raft_consensus.cc:2781] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.595475 16293 raft_consensus.cc:686] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:09.595736 16293 consensus_queue.cc:237] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:09.598126 12584 catalog_manager.cc:5484] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:09.607254 15755 raft_consensus.cc:3037] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.609462 16321 consensus_queue.cc:1035] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.614141 16341 consensus_queue.cc:1035] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.616643 16297 consensus_queue.cc:1035] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.622834 16298 consensus_queue.cc:1035] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.707509 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:09.712549 15755 raft_consensus.cc:1261] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:09.712592 15488 raft_consensus.cc:1261] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:09.713107 16293 consensus_queue.cc:1035] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.713413 16338 consensus_queue.cc:1035] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.717612 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet be2e95abf26f4c8284805524557079cd (table rebalancer_dynamic_table [id=9ffd254d93bc4b49884083efe9c8d8f0]). Expected version 1 got 0
I20240502 08:18:09.717869 16317 tablet.cc:1722] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:09.718137 12586 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet be2e95abf26f4c8284805524557079cd (table rebalancer_dynamic_table [id=9ffd254d93bc4b49884083efe9c8d8f0]). Expected version 1 got 0
I20240502 08:18:09.718364 16317 tablet.cc:1620] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:09.719485 16273 tablet.cc:1722] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:09.719941 16273 tablet.cc:1620] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:09.725461 12586 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=9ffd254d93bc4b49884083efe9c8d8f0] alter complete (version 1)
I20240502 08:18:09.729889 16316 tablet.cc:1722] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:09.730345 16316 tablet.cc:1620] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:09.750834 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:09.757817 15733 tablet_service.cc:1467] Processing CreateTablet for tablet be07b003a77249dc9a54243d1bfa2c50 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4b4d8bad8ac34c61b28c021c55674598]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.758066 15866 tablet_service.cc:1467] Processing CreateTablet for tablet be07b003a77249dc9a54243d1bfa2c50 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4b4d8bad8ac34c61b28c021c55674598]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.758199 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be07b003a77249dc9a54243d1bfa2c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.758483 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be07b003a77249dc9a54243d1bfa2c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.758831 15466 tablet_service.cc:1467] Processing CreateTablet for tablet 106ebc4b0ddd4acc824ae37042907a89 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4b4d8bad8ac34c61b28c021c55674598]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.758939 12970 tablet_service.cc:1467] Processing CreateTablet for tablet be07b003a77249dc9a54243d1bfa2c50 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4b4d8bad8ac34c61b28c021c55674598]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.759194 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106ebc4b0ddd4acc824ae37042907a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.759279 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be07b003a77249dc9a54243d1bfa2c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.762964 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 106ebc4b0ddd4acc824ae37042907a89 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4b4d8bad8ac34c61b28c021c55674598]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.763314 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106ebc4b0ddd4acc824ae37042907a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.764503 12840 tablet_service.cc:1467] Processing CreateTablet for tablet cb05ad19b6a84f35a6853db5e3cabf9a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4b4d8bad8ac34c61b28c021c55674598]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.764894 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb05ad19b6a84f35a6853db5e3cabf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.766594 16332 tablet_bootstrap.cc:492] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:09.766835 15600 tablet_service.cc:1467] Processing CreateTablet for tablet cb05ad19b6a84f35a6853db5e3cabf9a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4b4d8bad8ac34c61b28c021c55674598]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.767246 15600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb05ad19b6a84f35a6853db5e3cabf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.768237 16332 tablet_bootstrap.cc:654] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.769611 16334 tablet_bootstrap.cc:492] T be07b003a77249dc9a54243d1bfa2c50 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:09.771062 16334 tablet_bootstrap.cc:654] T be07b003a77249dc9a54243d1bfa2c50 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.771025 15865 tablet_service.cc:1467] Processing CreateTablet for tablet cb05ad19b6a84f35a6853db5e3cabf9a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4b4d8bad8ac34c61b28c021c55674598]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.771374 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb05ad19b6a84f35a6853db5e3cabf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.775384 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 106ebc4b0ddd4acc824ae37042907a89 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=4b4d8bad8ac34c61b28c021c55674598]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.775766 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106ebc4b0ddd4acc824ae37042907a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.780993 16348 tablet_bootstrap.cc:492] T be07b003a77249dc9a54243d1bfa2c50 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:09.784039 16348 tablet_bootstrap.cc:654] T be07b003a77249dc9a54243d1bfa2c50 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.785012 16333 tablet_bootstrap.cc:492] T cb05ad19b6a84f35a6853db5e3cabf9a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:09.786535 16333 tablet_bootstrap.cc:654] T cb05ad19b6a84f35a6853db5e3cabf9a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.789505 16348 tablet_bootstrap.cc:492] T be07b003a77249dc9a54243d1bfa2c50 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:09.789736 16348 ts_tablet_manager.cc:1367] T be07b003a77249dc9a54243d1bfa2c50 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 08:18:09.790583 16348 raft_consensus.cc:348] T be07b003a77249dc9a54243d1bfa2c50 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:09.790877 16348 raft_consensus.cc:374] T be07b003a77249dc9a54243d1bfa2c50 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.791002 16348 raft_consensus.cc:729] T be07b003a77249dc9a54243d1bfa2c50 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.791231 16348 consensus_queue.cc:260] T be07b003a77249dc9a54243d1bfa2c50 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:09.792353 16348 ts_tablet_manager.cc:1398] T be07b003a77249dc9a54243d1bfa2c50 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:09.792487 16334 tablet_bootstrap.cc:492] T be07b003a77249dc9a54243d1bfa2c50 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:09.792642 16334 ts_tablet_manager.cc:1367] T be07b003a77249dc9a54243d1bfa2c50 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.009s
I20240502 08:18:09.793362 16334 raft_consensus.cc:348] T be07b003a77249dc9a54243d1bfa2c50 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:09.793489 16333 tablet_bootstrap.cc:492] T cb05ad19b6a84f35a6853db5e3cabf9a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:09.793569 16334 raft_consensus.cc:374] T be07b003a77249dc9a54243d1bfa2c50 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.793637 16334 raft_consensus.cc:729] T be07b003a77249dc9a54243d1bfa2c50 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.793669 16333 ts_tablet_manager.cc:1367] T cb05ad19b6a84f35a6853db5e3cabf9a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240502 08:18:09.793834 16334 consensus_queue.cc:260] T be07b003a77249dc9a54243d1bfa2c50 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:09.793990 16352 tablet_bootstrap.cc:492] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:09.794371 16334 ts_tablet_manager.cc:1398] T be07b003a77249dc9a54243d1bfa2c50 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:09.795171 16332 tablet_bootstrap.cc:492] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:09.795365 16332 ts_tablet_manager.cc:1367] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.011s
I20240502 08:18:09.795415 16352 tablet_bootstrap.cc:654] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.795872 16333 raft_consensus.cc:348] T cb05ad19b6a84f35a6853db5e3cabf9a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:09.796135 16333 raft_consensus.cc:374] T cb05ad19b6a84f35a6853db5e3cabf9a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.796097 16332 raft_consensus.cc:348] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:09.796337 16332 raft_consensus.cc:374] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.796258 16333 raft_consensus.cc:729] T cb05ad19b6a84f35a6853db5e3cabf9a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.796422 16332 raft_consensus.cc:729] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.796598 16351 tablet_bootstrap.cc:492] T cb05ad19b6a84f35a6853db5e3cabf9a P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:09.796648 16332 consensus_queue.cc:260] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:09.796770 16333 consensus_queue.cc:260] T cb05ad19b6a84f35a6853db5e3cabf9a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:09.797147 16332 ts_tablet_manager.cc:1398] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:09.799525 16351 tablet_bootstrap.cc:654] T cb05ad19b6a84f35a6853db5e3cabf9a P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.799897 16333 ts_tablet_manager.cc:1398] T cb05ad19b6a84f35a6853db5e3cabf9a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:18:09.801636 16334 tablet_bootstrap.cc:492] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:09.803805 16334 tablet_bootstrap.cc:654] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.805398 16352 tablet_bootstrap.cc:492] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:09.805532 16334 tablet_bootstrap.cc:492] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:09.805663 16334 ts_tablet_manager.cc:1367] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:09.805764 16352 ts_tablet_manager.cc:1367] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240502 08:18:09.806524 16334 raft_consensus.cc:348] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:09.806706 16334 raft_consensus.cc:374] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.806793 16334 raft_consensus.cc:729] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.806582 16352 raft_consensus.cc:348] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:09.806937 16352 raft_consensus.cc:374] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.806950 16351 tablet_bootstrap.cc:492] T cb05ad19b6a84f35a6853db5e3cabf9a P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:09.807070 16352 raft_consensus.cc:729] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.807157 16351 ts_tablet_manager.cc:1367] T cb05ad19b6a84f35a6853db5e3cabf9a P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 08:18:09.807070 16334 consensus_queue.cc:260] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:09.807688 16334 ts_tablet_manager.cc:1398] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:09.807896 16352 consensus_queue.cc:260] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:09.808009 16351 raft_consensus.cc:348] T cb05ad19b6a84f35a6853db5e3cabf9a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:09.808228 16351 raft_consensus.cc:374] T cb05ad19b6a84f35a6853db5e3cabf9a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.808357 16351 raft_consensus.cc:729] T cb05ad19b6a84f35a6853db5e3cabf9a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.808621 16351 consensus_queue.cc:260] T cb05ad19b6a84f35a6853db5e3cabf9a P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:09.809178 16352 ts_tablet_manager.cc:1398] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:18:09.809283 16351 ts_tablet_manager.cc:1398] T cb05ad19b6a84f35a6853db5e3cabf9a P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:09.809481 16352 tablet_bootstrap.cc:492] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:09.810714 16352 tablet_bootstrap.cc:654] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.812391 16352 tablet_bootstrap.cc:492] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:09.812544 16352 ts_tablet_manager.cc:1367] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:09.813263 16352 raft_consensus.cc:348] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:09.813516 16352 raft_consensus.cc:374] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.813644 16352 raft_consensus.cc:729] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.813936 16352 consensus_queue.cc:260] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:09.815407 16352 ts_tablet_manager.cc:1398] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:09.815706 16352 tablet_bootstrap.cc:492] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:09.817353 16352 tablet_bootstrap.cc:654] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.819288 16352 tablet_bootstrap.cc:492] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:09.819423 16352 ts_tablet_manager.cc:1367] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 08:18:09.820006 16352 raft_consensus.cc:348] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:09.820142 16352 raft_consensus.cc:374] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.820240 16352 raft_consensus.cc:729] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.820428 16352 consensus_queue.cc:260] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:09.820863 16352 ts_tablet_manager.cc:1398] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:18:09.827453 12586 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:09.834728 16317 tablet.cc:1722] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:09.835187 16317 tablet.cc:1620] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:09.835327 16316 tablet.cc:1722] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:09.835716 16316 tablet.cc:1620] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:09.839711 16273 tablet.cc:1722] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:09.840162 16273 tablet.cc:1620] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:09.845371 12586 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=9ffd254d93bc4b49884083efe9c8d8f0] alter complete (version 2)
I20240502 08:18:09.872524 16024 raft_consensus.cc:482] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.872664 16024 raft_consensus.cc:504] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:09.873317 16024 leader_election.cc:288] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:09.873703 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be07b003a77249dc9a54243d1bfa2c50" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:09.873917 15755 raft_consensus.cc:2454] T be07b003a77249dc9a54243d1bfa2c50 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:09.875638 15818 leader_election.cc:302] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:09.875929 16024 raft_consensus.cc:2781] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.876052 16024 raft_consensus.cc:482] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.876154 16024 raft_consensus.cc:3037] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.878701 16024 raft_consensus.cc:504] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:09.879192 16024 leader_election.cc:288] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:09.879528 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be07b003a77249dc9a54243d1bfa2c50" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:09.879730 15755 raft_consensus.cc:3037] T be07b003a77249dc9a54243d1bfa2c50 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.880829 16024 raft_consensus.cc:482] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.880959 16024 raft_consensus.cc:504] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:09.881470 12991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be07b003a77249dc9a54243d1bfa2c50" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:09.881531 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be07b003a77249dc9a54243d1bfa2c50" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:09.881704 12991 raft_consensus.cc:2454] T be07b003a77249dc9a54243d1bfa2c50 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:09.881915 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05ad19b6a84f35a6853db5e3cabf9a" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:09.882104 15621 raft_consensus.cc:2454] T cb05ad19b6a84f35a6853db5e3cabf9a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:09.882164 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05ad19b6a84f35a6853db5e3cabf9a" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:09.882367 12861 raft_consensus.cc:2454] T cb05ad19b6a84f35a6853db5e3cabf9a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:09.882541 15822 leader_election.cc:302] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:09.882824 16221 raft_consensus.cc:2781] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.882922 16221 raft_consensus.cc:482] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.882992 16221 raft_consensus.cc:3037] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.883105 15755 raft_consensus.cc:2454] T be07b003a77249dc9a54243d1bfa2c50 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:09.883250 16024 leader_election.cc:288] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:09.883496 15818 leader_election.cc:302] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters: 60b6e265e7754b7187307d05ef405dd4
I20240502 08:18:09.883900 16024 raft_consensus.cc:2781] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.884346 16024 raft_consensus.cc:686] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:09.884678 16024 consensus_queue.cc:237] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:09.885452 16221 raft_consensus.cc:504] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:09.886446 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05ad19b6a84f35a6853db5e3cabf9a" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:09.886446 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05ad19b6a84f35a6853db5e3cabf9a" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:09.886659 15621 raft_consensus.cc:3037] T cb05ad19b6a84f35a6853db5e3cabf9a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.886659 12861 raft_consensus.cc:3037] T cb05ad19b6a84f35a6853db5e3cabf9a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.889273 15621 raft_consensus.cc:2454] T cb05ad19b6a84f35a6853db5e3cabf9a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:09.889273 12861 raft_consensus.cc:2454] T cb05ad19b6a84f35a6853db5e3cabf9a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:09.889614 15819 leader_election.cc:302] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:09.889804 16221 leader_election.cc:288] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:09.889967 16024 raft_consensus.cc:2781] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.890079 16024 raft_consensus.cc:686] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:09.890375 16024 consensus_queue.cc:237] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:09.890596 12586 catalog_manager.cc:5484] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:09.894673 12586 catalog_manager.cc:5484] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:09.948597 12586 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:09.955024 16317 tablet.cc:1722] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:09.955629 16316 tablet.cc:1722] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:09.955698 16273 tablet.cc:1722] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:09.963518 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=9ffd254d93bc4b49884083efe9c8d8f0] alter complete (version 3)
I20240502 08:18:10.011735 16298 raft_consensus.cc:482] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.011904 16298 raft_consensus.cc:504] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.012495 16298 leader_election.cc:288] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:10.012931 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106ebc4b0ddd4acc824ae37042907a89" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:10.012954 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106ebc4b0ddd4acc824ae37042907a89" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:10.013175 15488 raft_consensus.cc:2454] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:10.013202 15886 raft_consensus.cc:2454] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:10.013615 15688 leader_election.cc:302] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:10.013890 16298 raft_consensus.cc:2781] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.014011 16298 raft_consensus.cc:482] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.014092 16298 raft_consensus.cc:3037] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.016810 16298 raft_consensus.cc:504] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.017432 16298 leader_election.cc:288] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:10.017768 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106ebc4b0ddd4acc824ae37042907a89" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:10.017868 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106ebc4b0ddd4acc824ae37042907a89" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:10.017969 15886 raft_consensus.cc:3037] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.018057 15488 raft_consensus.cc:3037] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.020399 15488 raft_consensus.cc:2454] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:10.020560 15886 raft_consensus.cc:2454] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:10.020735 15688 leader_election.cc:302] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:10.021008 16298 raft_consensus.cc:2781] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:10.021157 16298 raft_consensus.cc:686] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:10.021411 16298 consensus_queue.cc:237] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.024226 12586 catalog_manager.cc:5484] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:10.065531 12586 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:10.065804 12586 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:10.068764 12586 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet be2e95abf26f4c8284805524557079cd
I20240502 08:18:10.069502 15735 tablet_service.cc:1514] Processing DeleteTablet for tablet be2e95abf26f4c8284805524557079cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:10 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:10.069711 15466 tablet_service.cc:1514] Processing DeleteTablet for tablet be2e95abf26f4c8284805524557079cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:10 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:10.070075 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet be2e95abf26f4c8284805524557079cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:10 UTC) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:10.070192 16367 tablet_replica.cc:330] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:10.070238 16368 tablet_replica.cc:330] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:10.070430 16367 raft_consensus.cc:2229] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:10.070477 16368 raft_consensus.cc:2229] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:10.070676 16369 tablet_replica.cc:330] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:10.070719 16367 raft_consensus.cc:2258] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:10.070732 16368 raft_consensus.cc:2258] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:10.070931 16369 raft_consensus.cc:2229] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:10.071310 16369 raft_consensus.cc:2258] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:10.071529 16367 ts_tablet_manager.cc:1875] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:10.071578 16368 ts_tablet_manager.cc:1875] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:10.072055 16369 ts_tablet_manager.cc:1875] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:10.076288 16367 ts_tablet_manager.cc:1888] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:10.076393 16367 log.cc:1196] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/be2e95abf26f4c8284805524557079cd
I20240502 08:18:10.076786 16367 ts_tablet_manager.cc:1909] T be2e95abf26f4c8284805524557079cd P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:10.077749 16368 ts_tablet_manager.cc:1888] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:10.077744 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet be2e95abf26f4c8284805524557079cd (table rebalancer_dynamic_table_new_name [id=9ffd254d93bc4b49884083efe9c8d8f0]) successfully deleted
I20240502 08:18:10.077901 16368 log.cc:1196] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/be2e95abf26f4c8284805524557079cd
I20240502 08:18:10.077916 16369 ts_tablet_manager.cc:1888] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:10.078013 16369 log.cc:1196] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/be2e95abf26f4c8284805524557079cd
I20240502 08:18:10.078356 16368 ts_tablet_manager.cc:1909] T be2e95abf26f4c8284805524557079cd P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:10.078395 16369 ts_tablet_manager.cc:1909] T be2e95abf26f4c8284805524557079cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting consensus metadata
I20240502 08:18:10.079324 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet be2e95abf26f4c8284805524557079cd (table rebalancer_dynamic_table_new_name [id=9ffd254d93bc4b49884083efe9c8d8f0]) successfully deleted
I20240502 08:18:10.079324 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet be2e95abf26f4c8284805524557079cd (table rebalancer_dynamic_table_new_name [id=9ffd254d93bc4b49884083efe9c8d8f0]) successfully deleted
I20240502 08:18:10.171200 12586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:10.178732 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed73c6bebb34913b3ab2baee265ef58 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4911813f12644545b58c8cf658723ce2]), partition=
I20240502 08:18:10.179068 15466 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed73c6bebb34913b3ab2baee265ef58 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4911813f12644545b58c8cf658723ce2]), partition=
I20240502 08:18:10.179162 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed73c6bebb34913b3ab2baee265ef58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.179415 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed73c6bebb34913b3ab2baee265ef58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.183840 16352 tablet_bootstrap.cc:492] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:10.184953 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed73c6bebb34913b3ab2baee265ef58 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4911813f12644545b58c8cf658723ce2]), partition=
I20240502 08:18:10.185295 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed73c6bebb34913b3ab2baee265ef58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.185696 16352 tablet_bootstrap.cc:654] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.187693 16352 tablet_bootstrap.cc:492] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:10.187844 16332 tablet_bootstrap.cc:492] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:10.187876 16352 ts_tablet_manager.cc:1367] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:10.188673 16352 raft_consensus.cc:348] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.189405 16332 tablet_bootstrap.cc:654] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.189694 16334 tablet_bootstrap.cc:492] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:10.188897 16352 raft_consensus.cc:374] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.190491 16352 raft_consensus.cc:729] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.190853 16352 consensus_queue.cc:260] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.191087 16334 tablet_bootstrap.cc:654] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.191413 16352 ts_tablet_manager.cc:1398] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 08:18:10.200979 16332 tablet_bootstrap.cc:492] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:10.201259 16334 tablet_bootstrap.cc:492] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:10.201443 16332 ts_tablet_manager.cc:1367] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20240502 08:18:10.201455 16334 ts_tablet_manager.cc:1367] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20240502 08:18:10.202181 16334 raft_consensus.cc:348] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.202204 16332 raft_consensus.cc:348] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.202440 16334 raft_consensus.cc:374] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.202555 16332 raft_consensus.cc:374] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.202572 16334 raft_consensus.cc:729] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.202761 16332 raft_consensus.cc:729] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.202785 16334 consensus_queue.cc:260] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.203307 16334 ts_tablet_manager.cc:1398] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:10.203007 16332 consensus_queue.cc:260] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.204071 16332 ts_tablet_manager.cc:1398] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:10.262223 16341 raft_consensus.cc:482] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.262395 16341 raft_consensus.cc:504] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.262987 16341 leader_election.cc:288] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:10.263271 16024 raft_consensus.cc:482] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.263422 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed73c6bebb34913b3ab2baee265ef58" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:10.263484 16024 raft_consensus.cc:504] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.263522 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed73c6bebb34913b3ab2baee265ef58" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:10.263742 15755 raft_consensus.cc:2454] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:10.263797 15886 raft_consensus.cc:2454] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:10.264200 16024 leader_election.cc:288] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:10.264189 15418 leader_election.cc:302] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:10.264637 16341 raft_consensus.cc:2781] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.264581 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed73c6bebb34913b3ab2baee265ef58" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:10.264745 16341 raft_consensus.cc:482] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.264838 16341 raft_consensus.cc:3037] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.264822 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed73c6bebb34913b3ab2baee265ef58" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:10.265029 15755 raft_consensus.cc:2454] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:10.265403 15818 leader_election.cc:302] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:10.265753 16024 raft_consensus.cc:2781] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.265851 16024 raft_consensus.cc:482] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.265964 16024 raft_consensus.cc:3037] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.267632 16341 raft_consensus.cc:504] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.268265 16341 leader_election.cc:288] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:10.268723 15488 raft_consensus.cc:2379] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10f0e36203914b85b77e78609ffced98 in current term 1: Already voted for candidate db7893bb29a047b3b82cc60bfd8930a3 in this term.
I20240502 08:18:10.268967 16024 raft_consensus.cc:504] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.269097 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed73c6bebb34913b3ab2baee265ef58" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:10.269361 15886 raft_consensus.cc:2379] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate db7893bb29a047b3b82cc60bfd8930a3 in current term 1: Already voted for candidate 10f0e36203914b85b77e78609ffced98 in this term.
I20240502 08:18:10.270002 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed73c6bebb34913b3ab2baee265ef58" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:10.270164 15755 raft_consensus.cc:3037] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.270191 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed73c6bebb34913b3ab2baee265ef58" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:10.270409 15488 raft_consensus.cc:2379] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10f0e36203914b85b77e78609ffced98 in current term 1: Already voted for candidate db7893bb29a047b3b82cc60bfd8930a3 in this term.
I20240502 08:18:10.270524 16024 leader_election.cc:288] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:10.271205 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed73c6bebb34913b3ab2baee265ef58" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:10.273375 15755 raft_consensus.cc:2454] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:10.274056 15818 leader_election.cc:302] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10f0e36203914b85b77e78609ffced98; no voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:10.274394 16024 raft_consensus.cc:2726] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:18:10.274369 15418 leader_election.cc:302] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters: 10f0e36203914b85b77e78609ffced98
I20240502 08:18:10.274742 16341 raft_consensus.cc:2781] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:10.275115 16341 raft_consensus.cc:686] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:10.275575 16341 consensus_queue.cc:237] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.278134 12586 catalog_manager.cc:5484] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:10.315688 12992 raft_consensus.cc:3037] T be07b003a77249dc9a54243d1bfa2c50 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.321681 16024 consensus_queue.cc:1035] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.326577 16024 consensus_queue.cc:1035] T be07b003a77249dc9a54243d1bfa2c50 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.394551 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:10.399463 15755 raft_consensus.cc:1261] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:10.399495 15886 raft_consensus.cc:1261] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:10.400002 16373 consensus_queue.cc:1035] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.400300 16341 consensus_queue.cc:1035] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.404068 12586 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet 1ed73c6bebb34913b3ab2baee265ef58 (table rebalancer_dynamic_table [id=4911813f12644545b58c8cf658723ce2]). Expected version 1 got 0
I20240502 08:18:10.404214 16273 tablet.cc:1722] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:10.404625 16273 tablet.cc:1620] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:10.404820 16221 consensus_queue.cc:1035] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.407423 12584 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 1ed73c6bebb34913b3ab2baee265ef58 (table rebalancer_dynamic_table [id=4911813f12644545b58c8cf658723ce2]). Expected version 1 got 0
I20240502 08:18:10.409554 16316 tablet.cc:1722] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:10.409967 16316 tablet.cc:1620] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:10.412575 16362 consensus_queue.cc:1035] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.413022 12583 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet 1ed73c6bebb34913b3ab2baee265ef58 (table rebalancer_dynamic_table [id=4911813f12644545b58c8cf658723ce2]). Expected version 1 got 0
I20240502 08:18:10.415807 16382 tablet.cc:1722] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:10.416361 16382 tablet.cc:1620] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:10.419133 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=4911813f12644545b58c8cf658723ce2] alter complete (version 1)
I20240502 08:18:10.427910 16298 consensus_queue.cc:1035] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.432757 16298 consensus_queue.cc:1035] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.523324 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:10.530176 16273 tablet.cc:1722] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:10.530711 16273 tablet.cc:1620] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:10.530933 16316 tablet.cc:1722] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:10.531257 16382 tablet.cc:1722] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:10.531364 16316 tablet.cc:1620] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:10.531669 16382 tablet.cc:1620] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:10.538807 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=4911813f12644545b58c8cf658723ce2] alter complete (version 2)
I20240502 08:18:10.569355 12587 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:10.577447 15466 tablet_service.cc:1467] Processing CreateTablet for tablet 95d1c7bd2c424900b399fac3fdb5c9ac (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e77588602bd34d11a274be948cc9d52a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.577643 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 95d1c7bd2c424900b399fac3fdb5c9ac (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e77588602bd34d11a274be948cc9d52a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.577947 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d1c7bd2c424900b399fac3fdb5c9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.578029 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d1c7bd2c424900b399fac3fdb5c9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.577929 15600 tablet_service.cc:1467] Processing CreateTablet for tablet 95d1c7bd2c424900b399fac3fdb5c9ac (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e77588602bd34d11a274be948cc9d52a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.578369 15600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d1c7bd2c424900b399fac3fdb5c9ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.578806 15601 tablet_service.cc:1467] Processing CreateTablet for tablet c2ad585a32864067a5d8eaea5f8260f1 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e77588602bd34d11a274be948cc9d52a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.579139 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ad585a32864067a5d8eaea5f8260f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.584295 16334 tablet_bootstrap.cc:492] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:10.585197 12840 tablet_service.cc:1467] Processing CreateTablet for tablet c2ad585a32864067a5d8eaea5f8260f1 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e77588602bd34d11a274be948cc9d52a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.585453 16388 tablet_bootstrap.cc:492] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:10.585595 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ad585a32864067a5d8eaea5f8260f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.585824 16334 tablet_bootstrap.cc:654] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.586727 16332 tablet_bootstrap.cc:492] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:10.587414 16388 tablet_bootstrap.cc:654] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.587874 16334 tablet_bootstrap.cc:492] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:10.587975 16334 ts_tablet_manager.cc:1367] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:10.588281 16332 tablet_bootstrap.cc:654] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.588618 16334 raft_consensus.cc:348] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.588829 16334 raft_consensus.cc:374] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.588915 16334 raft_consensus.cc:729] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.589143 16334 consensus_queue.cc:260] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.590152 16334 ts_tablet_manager.cc:1398] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:10.591555 16388 tablet_bootstrap.cc:492] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:10.591711 16388 ts_tablet_manager.cc:1367] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:18:10.592414 16388 raft_consensus.cc:348] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.592618 16388 raft_consensus.cc:374] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.592710 16388 raft_consensus.cc:729] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.592974 16388 consensus_queue.cc:260] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.593508 16388 ts_tablet_manager.cc:1398] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:10.593756 16388 tablet_bootstrap.cc:492] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:10.595247 16388 tablet_bootstrap.cc:654] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.596215 15735 tablet_service.cc:1467] Processing CreateTablet for tablet c2ad585a32864067a5d8eaea5f8260f1 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e77588602bd34d11a274be948cc9d52a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.596457 16332 tablet_bootstrap.cc:492] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:10.596599 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ad585a32864067a5d8eaea5f8260f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.596621 16332 ts_tablet_manager.cc:1367] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20240502 08:18:10.597311 16332 raft_consensus.cc:348] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.597545 16332 raft_consensus.cc:374] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.597617 16332 raft_consensus.cc:729] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.597836 16332 consensus_queue.cc:260] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.598322 16332 ts_tablet_manager.cc:1398] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:10.602522 16393 tablet_bootstrap.cc:492] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:10.602788 15867 tablet_service.cc:1467] Processing CreateTablet for tablet ad1bea423b354df38e9e0100c1560e08 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e77588602bd34d11a274be948cc9d52a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.603209 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1bea423b354df38e9e0100c1560e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.604409 16393 tablet_bootstrap.cc:654] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.606935 15733 tablet_service.cc:1467] Processing CreateTablet for tablet ad1bea423b354df38e9e0100c1560e08 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e77588602bd34d11a274be948cc9d52a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.607336 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1bea423b354df38e9e0100c1560e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.608007 16334 tablet_bootstrap.cc:492] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:10.611182 16393 tablet_bootstrap.cc:492] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:10.611367 16393 ts_tablet_manager.cc:1367] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 08:18:10.612169 16393 raft_consensus.cc:348] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.611497 15601 tablet_service.cc:1467] Processing CreateTablet for tablet ad1bea423b354df38e9e0100c1560e08 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=e77588602bd34d11a274be948cc9d52a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.612442 16393 raft_consensus.cc:374] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.612540 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1bea423b354df38e9e0100c1560e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.612576 16393 raft_consensus.cc:729] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.612807 16393 consensus_queue.cc:260] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.613493 16393 ts_tablet_manager.cc:1398] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:10.616595 16352 tablet_bootstrap.cc:492] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:10.618167 16352 tablet_bootstrap.cc:654] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.619920 16352 tablet_bootstrap.cc:492] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:10.620051 16352 ts_tablet_manager.cc:1367] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:10.620719 16352 raft_consensus.cc:348] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.620908 16352 raft_consensus.cc:374] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.620987 16352 raft_consensus.cc:729] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.621191 16352 consensus_queue.cc:260] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.621687 16352 ts_tablet_manager.cc:1398] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:10.630640 16334 tablet_bootstrap.cc:654] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.632308 16334 tablet_bootstrap.cc:492] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:10.632472 16334 ts_tablet_manager.cc:1367] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.002s
I20240502 08:18:10.633112 16334 raft_consensus.cc:348] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.633363 16334 raft_consensus.cc:374] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.633435 16334 raft_consensus.cc:729] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.633667 16334 consensus_queue.cc:260] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.634301 16334 ts_tablet_manager.cc:1398] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:10.634626 16334 tablet_bootstrap.cc:492] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:10.636163 16334 tablet_bootstrap.cc:654] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.639169 16388 tablet_bootstrap.cc:492] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:10.639322 16388 ts_tablet_manager.cc:1367] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.006s
I20240502 08:18:10.639894 16388 raft_consensus.cc:348] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.640106 16388 raft_consensus.cc:374] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.640173 16388 raft_consensus.cc:729] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.640388 16388 consensus_queue.cc:260] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.640885 16388 ts_tablet_manager.cc:1398] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:18:10.641185 16388 tablet_bootstrap.cc:492] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:10.644397 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:10.649392 16334 tablet_bootstrap.cc:492] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:10.649533 16334 ts_tablet_manager.cc:1367] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20240502 08:18:10.650236 16334 raft_consensus.cc:348] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.650419 16334 raft_consensus.cc:374] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.650570 16334 raft_consensus.cc:729] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.650785 16334 consensus_queue.cc:260] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.651937 16334 ts_tablet_manager.cc:1398] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:10.659219 16388 tablet_bootstrap.cc:654] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.660195 16273 tablet.cc:1722] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:10.661327 16316 tablet.cc:1722] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:10.664496 16382 tablet.cc:1722] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:10.666671 16024 raft_consensus.cc:482] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.666867 16024 raft_consensus.cc:504] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.667591 16024 leader_election.cc:288] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:10.668030 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1bea423b354df38e9e0100c1560e08" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:10.668437 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1bea423b354df38e9e0100c1560e08" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:10.668622 15755 raft_consensus.cc:2454] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
W20240502 08:18:10.668632 15822 leader_election.cc:341] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:10.668978 15818 leader_election.cc:302] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:10.669243 16024 raft_consensus.cc:2781] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.669355 16024 raft_consensus.cc:482] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.669427 16024 raft_consensus.cc:3037] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.671386 16024 raft_consensus.cc:504] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.672628 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1bea423b354df38e9e0100c1560e08" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:10.672814 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=4911813f12644545b58c8cf658723ce2] alter complete (version 3)
W20240502 08:18:10.673157 15822 leader_election.cc:341] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:10.673779 16024 leader_election.cc:288] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:10.673892 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1bea423b354df38e9e0100c1560e08" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:10.674057 15755 raft_consensus.cc:3037] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.674511 16258 raft_consensus.cc:482] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.674659 16258 raft_consensus.cc:504] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.675509 16258 leader_election.cc:288] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:10.676759 15755 raft_consensus.cc:2454] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:10.677101 15818 leader_election.cc:302] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:10.677399 16024 raft_consensus.cc:2781] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:10.677563 16024 raft_consensus.cc:686] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:10.677892 16024 consensus_queue.cc:237] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.680011 16388 tablet_bootstrap.cc:492] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:10.680200 16388 ts_tablet_manager.cc:1367] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20240502 08:18:10.681003 12587 catalog_manager.cc:5484] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:10.681131 16388 raft_consensus.cc:348] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.681407 16388 raft_consensus.cc:374] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.681501 16388 raft_consensus.cc:729] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.681735 16388 consensus_queue.cc:260] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.683483 16388 ts_tablet_manager.cc:1398] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:10.686017 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ad585a32864067a5d8eaea5f8260f1" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:10.686280 15621 raft_consensus.cc:2454] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:10.686868 12798 leader_election.cc:302] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:10.687134 16258 raft_consensus.cc:2781] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.687237 16258 raft_consensus.cc:482] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.687319 16258 raft_consensus.cc:3037] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.690114 16258 raft_consensus.cc:504] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.690985 16258 leader_election.cc:288] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:10.691888 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ad585a32864067a5d8eaea5f8260f1" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:10.692083 15621 raft_consensus.cc:3037] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.694653 15621 raft_consensus.cc:2454] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:10.695015 12798 leader_election.cc:302] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:10.695451 16258 raft_consensus.cc:2781] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:10.695894 16258 raft_consensus.cc:686] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:18:10.696218 16258 consensus_queue.cc:237] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:10.700182 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ad585a32864067a5d8eaea5f8260f1" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:10.700397 15755 raft_consensus.cc:2454] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:10.701052 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ad585a32864067a5d8eaea5f8260f1" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:10.701217 15755 raft_consensus.cc:3037] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.701074 12583 catalog_manager.cc:5484] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:10.704090 15755 raft_consensus.cc:2454] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:10.777870 12583 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:10.778092 12583 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:10.781037 12583 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 1ed73c6bebb34913b3ab2baee265ef58
I20240502 08:18:10.781764 15867 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed73c6bebb34913b3ab2baee265ef58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:10 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:10.781930 15466 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed73c6bebb34913b3ab2baee265ef58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:10 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:10.782368 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed73c6bebb34913b3ab2baee265ef58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:10 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:10.782502 16406 tablet_replica.cc:330] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:10.782776 16406 raft_consensus.cc:2229] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:10.783411 16406 raft_consensus.cc:2258] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:10.783771 16407 tablet_replica.cc:330] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:10.783938 16408 tablet_replica.cc:330] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:10.784010 16407 raft_consensus.cc:2229] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:10.784153 16408 raft_consensus.cc:2229] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:10.784389 16408 raft_consensus.cc:2258] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:10.784404 16407 raft_consensus.cc:2258] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:10.784555 16406 ts_tablet_manager.cc:1875] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:10.785218 16407 ts_tablet_manager.cc:1875] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:10.785219 16408 ts_tablet_manager.cc:1875] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:10.789575 16406 ts_tablet_manager.cc:1888] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:10.789701 16406 log.cc:1196] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/1ed73c6bebb34913b3ab2baee265ef58
I20240502 08:18:10.790196 16406 ts_tablet_manager.cc:1909] T 1ed73c6bebb34913b3ab2baee265ef58 P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:10.791224 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 1ed73c6bebb34913b3ab2baee265ef58 (table rebalancer_dynamic_table_new_name [id=4911813f12644545b58c8cf658723ce2]) successfully deleted
I20240502 08:18:10.791272 16408 ts_tablet_manager.cc:1888] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:10.791378 16408 log.cc:1196] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/1ed73c6bebb34913b3ab2baee265ef58
I20240502 08:18:10.791554 16407 ts_tablet_manager.cc:1888] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:10.791671 16407 log.cc:1196] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/1ed73c6bebb34913b3ab2baee265ef58
I20240502 08:18:10.791831 16408 ts_tablet_manager.cc:1909] T 1ed73c6bebb34913b3ab2baee265ef58 P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:10.792047 16407 ts_tablet_manager.cc:1909] T 1ed73c6bebb34913b3ab2baee265ef58 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:10.792865 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 1ed73c6bebb34913b3ab2baee265ef58 (table rebalancer_dynamic_table_new_name [id=4911813f12644545b58c8cf658723ce2]) successfully deleted
I20240502 08:18:10.792973 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 1ed73c6bebb34913b3ab2baee265ef58 (table rebalancer_dynamic_table_new_name [id=4911813f12644545b58c8cf658723ce2]) successfully deleted
I20240502 08:18:10.861610 16228 raft_consensus.cc:482] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.861719 16341 raft_consensus.cc:482] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.861801 16228 raft_consensus.cc:504] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.861873 16341 raft_consensus.cc:504] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.862491 16341 leader_election.cc:288] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:10.862491 16228 leader_election.cc:288] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:10.862860 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d1c7bd2c424900b399fac3fdb5c9ac" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:10.863056 15755 raft_consensus.cc:2454] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:10.862977 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d1c7bd2c424900b399fac3fdb5c9ac" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:10.863026 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d1c7bd2c424900b399fac3fdb5c9ac" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:10.863232 15621 raft_consensus.cc:2454] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:10.863564 15421 leader_election.cc:302] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:10.863879 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d1c7bd2c424900b399fac3fdb5c9ac" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:10.864069 15488 raft_consensus.cc:2454] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:10.864336 16341 raft_consensus.cc:2781] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.864423 16341 raft_consensus.cc:482] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.864425 15688 leader_election.cc:302] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:10.864516 16341 raft_consensus.cc:3037] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.864698 16228 raft_consensus.cc:2781] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.864785 16228 raft_consensus.cc:482] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.864902 16228 raft_consensus.cc:3037] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.867158 16341 raft_consensus.cc:504] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.867436 16228 raft_consensus.cc:504] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.867801 16341 leader_election.cc:288] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:10.868135 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d1c7bd2c424900b399fac3fdb5c9ac" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:10.868137 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d1c7bd2c424900b399fac3fdb5c9ac" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:10.868338 15755 raft_consensus.cc:2379] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate db7893bb29a047b3b82cc60bfd8930a3 in current term 1: Already voted for candidate 9333cb10227b4801b4fa7c24e22e131b in this term.
I20240502 08:18:10.868335 15621 raft_consensus.cc:3037] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.868511 16228 leader_election.cc:288] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:10.870182 15621 raft_consensus.cc:2454] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:10.870528 15421 leader_election.cc:302] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3; no voters: 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:10.870698 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d1c7bd2c424900b399fac3fdb5c9ac" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:10.870777 16341 raft_consensus.cc:2781] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:10.870905 15622 raft_consensus.cc:2379] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9333cb10227b4801b4fa7c24e22e131b in current term 1: Already voted for candidate db7893bb29a047b3b82cc60bfd8930a3 in this term.
I20240502 08:18:10.870858 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d1c7bd2c424900b399fac3fdb5c9ac" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:10.870932 16341 raft_consensus.cc:686] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:10.871186 16341 consensus_queue.cc:237] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.872740 15688 leader_election.cc:302] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:10.872988 16228 raft_consensus.cc:2726] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:18:10.873914 12583 catalog_manager.cc:5484] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:10.883728 12583 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:10.885614 16293 raft_consensus.cc:482] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.885759 16293 raft_consensus.cc:504] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:10.886281 16293 leader_election.cc:288] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:10.886633 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1bea423b354df38e9e0100c1560e08" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:10.886873 15755 raft_consensus.cc:2379] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in current term 1: Already voted for candidate 10f0e36203914b85b77e78609ffced98 in this term.
I20240502 08:18:10.886966 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1bea423b354df38e9e0100c1560e08" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:10.888720 15553 leader_election.cc:302] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:10.889076 16293 raft_consensus.cc:3037] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.891186 15466 tablet_service.cc:1467] Processing CreateTablet for tablet 421829e63d274a978e337607efb2cff3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=de593c2fc465447cb6c70e3db40a5e18]), partition=
I20240502 08:18:10.891525 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 421829e63d274a978e337607efb2cff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.891526 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 421829e63d274a978e337607efb2cff3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=de593c2fc465447cb6c70e3db40a5e18]), partition=
I20240502 08:18:10.891541 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 421829e63d274a978e337607efb2cff3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=de593c2fc465447cb6c70e3db40a5e18]), partition=
I20240502 08:18:10.891836 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 421829e63d274a978e337607efb2cff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.891872 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 421829e63d274a978e337607efb2cff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.896374 16293 raft_consensus.cc:2726] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:10.896868 16410 tablet_bootstrap.cc:492] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:10.897329 16332 tablet_bootstrap.cc:492] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:10.898813 16332 tablet_bootstrap.cc:654] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.899036 16410 tablet_bootstrap.cc:654] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.900609 16332 tablet_bootstrap.cc:492] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:10.900740 16332 ts_tablet_manager.cc:1367] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:10.900856 16410 tablet_bootstrap.cc:492] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:10.901176 16410 ts_tablet_manager.cc:1367] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:10.901436 16332 raft_consensus.cc:348] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:10.901568 16332 raft_consensus.cc:374] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.901621 16332 raft_consensus.cc:729] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.901803 16332 consensus_queue.cc:260] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:10.901924 16410 raft_consensus.cc:348] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:10.902122 16410 raft_consensus.cc:374] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.902207 16410 raft_consensus.cc:729] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.902299 16332 ts_tablet_manager.cc:1398] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:18:10.902397 16410 consensus_queue.cc:260] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:10.902969 16410 ts_tablet_manager.cc:1398] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:10.906553 16411 tablet_bootstrap.cc:492] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:10.907850 16411 tablet_bootstrap.cc:654] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.909471 16411 tablet_bootstrap.cc:492] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:10.909579 16411 ts_tablet_manager.cc:1367] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:10.910133 16411 raft_consensus.cc:348] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:10.910324 16411 raft_consensus.cc:374] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.910413 16411 raft_consensus.cc:729] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.910614 16411 consensus_queue.cc:260] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:10.911082 16411 ts_tablet_manager.cc:1398] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:18:11.030704 16187 raft_consensus.cc:482] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.030874 16187 raft_consensus.cc:504] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:11.031601 16187 leader_election.cc:288] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:11.032016 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "421829e63d274a978e337607efb2cff3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:11.032094 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "421829e63d274a978e337607efb2cff3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:11.032256 15488 raft_consensus.cc:2454] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:11.032338 12992 raft_consensus.cc:2454] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:11.032634 12664 leader_election.cc:302] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:11.032886 16187 raft_consensus.cc:2781] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.032996 16187 raft_consensus.cc:482] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.033094 16187 raft_consensus.cc:3037] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.035840 16187 raft_consensus.cc:504] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:11.036425 16187 leader_election.cc:288] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:11.036830 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "421829e63d274a978e337607efb2cff3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:11.036932 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "421829e63d274a978e337607efb2cff3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:11.037041 15488 raft_consensus.cc:3037] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.037145 12992 raft_consensus.cc:3037] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.039031 15488 raft_consensus.cc:2454] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:11.039484 12992 raft_consensus.cc:2454] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:11.039651 12664 leader_election.cc:302] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:11.039903 16187 raft_consensus.cc:2781] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.040305 16187 raft_consensus.cc:686] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:18:11.040565 16187 consensus_queue.cc:237] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:11.043295 12583 catalog_manager.cc:5484] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:11.147446 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:11.149168 16258 consensus_queue.cc:1035] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.154752 12992 raft_consensus.cc:1261] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.155368 15487 raft_consensus.cc:1261] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.155413 16241 consensus_queue.cc:1035] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.156139 16241 consensus_queue.cc:1035] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.164057 16425 tablet.cc:1722] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:11.165275 16404 consensus_queue.cc:1035] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.166373 16425 tablet.cc:1620] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:11.167209 16024 consensus_queue.cc:1035] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.171101 12583 catalog_manager.cc:5576] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757) does not have the latest schema for tablet 421829e63d274a978e337607efb2cff3 (table rebalancer_dynamic_table [id=de593c2fc465447cb6c70e3db40a5e18]). Expected version 1 got 0
I20240502 08:18:11.179558 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=de593c2fc465447cb6c70e3db40a5e18] alter complete (version 1)
I20240502 08:18:11.198312 16424 tablet.cc:1722] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:11.199913 16424 tablet.cc:1620] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:11.214871 16434 tablet.cc:1722] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:11.215584 16434 tablet.cc:1620] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:11.219393 16024 consensus_queue.cc:1035] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.225023 12587 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet 421829e63d274a978e337607efb2cff3 (table rebalancer_dynamic_table [id=de593c2fc465447cb6c70e3db40a5e18]). Expected version 1 got 0
I20240502 08:18:11.287873 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:11.295949 16424 tablet.cc:1722] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:11.296418 16424 tablet.cc:1620] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:11.296763 16434 tablet.cc:1722] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:11.296840 16425 tablet.cc:1722] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:11.297236 16425 tablet.cc:1620] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:11.297235 16434 tablet.cc:1620] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:11.307219 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=de593c2fc465447cb6c70e3db40a5e18] alter complete (version 2)
I20240502 08:18:11.339064 16341 consensus_queue.cc:1035] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.339295 16373 consensus_queue.cc:1035] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.386979 12587 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:11.395251 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 42e0e206eca841fd8dfa7146f8f2d890 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=69a26fefc8344774a5308e75413de1d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.395709 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e0e206eca841fd8dfa7146f8f2d890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.395690 15466 tablet_service.cc:1467] Processing CreateTablet for tablet 42e0e206eca841fd8dfa7146f8f2d890 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=69a26fefc8344774a5308e75413de1d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.396076 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e0e206eca841fd8dfa7146f8f2d890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.398847 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 5205c45461da48369a6f0d103f8a94e6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=69a26fefc8344774a5308e75413de1d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.399253 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5205c45461da48369a6f0d103f8a94e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.402076 16332 tablet_bootstrap.cc:492] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:11.402199 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 8774c037b901461684d96460e663d17e (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=69a26fefc8344774a5308e75413de1d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.402617 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8774c037b901461684d96460e663d17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.403115 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 5205c45461da48369a6f0d103f8a94e6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=69a26fefc8344774a5308e75413de1d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.403225 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 8774c037b901461684d96460e663d17e (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=69a26fefc8344774a5308e75413de1d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.403482 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5205c45461da48369a6f0d103f8a94e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.403666 16332 tablet_bootstrap.cc:654] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.403863 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8774c037b901461684d96460e663d17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.404909 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 42e0e206eca841fd8dfa7146f8f2d890 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=69a26fefc8344774a5308e75413de1d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.405287 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e0e206eca841fd8dfa7146f8f2d890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.407712 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 8774c037b901461684d96460e663d17e (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=69a26fefc8344774a5308e75413de1d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.408080 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8774c037b901461684d96460e663d17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.413412 15600 tablet_service.cc:1467] Processing CreateTablet for tablet 5205c45461da48369a6f0d103f8a94e6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=69a26fefc8344774a5308e75413de1d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:11.413789 15600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5205c45461da48369a6f0d103f8a94e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.413172 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:11.418653 16442 tablet_bootstrap.cc:492] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:11.418684 16440 tablet_bootstrap.cc:492] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:11.420171 16442 tablet_bootstrap.cc:654] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.421766 16440 tablet_bootstrap.cc:654] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.427131 16424 tablet.cc:1722] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:11.431510 16438 tablet_bootstrap.cc:492] T 42e0e206eca841fd8dfa7146f8f2d890 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:11.434568 16438 tablet_bootstrap.cc:654] T 42e0e206eca841fd8dfa7146f8f2d890 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.436004 16434 tablet.cc:1722] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:11.445742 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=de593c2fc465447cb6c70e3db40a5e18] alter complete (version 3)
I20240502 08:18:11.448334 16441 tablet_bootstrap.cc:492] T 42e0e206eca841fd8dfa7146f8f2d890 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:11.450269 16441 tablet_bootstrap.cc:654] T 42e0e206eca841fd8dfa7146f8f2d890 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.451773 16332 tablet_bootstrap.cc:492] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:11.452018 16332 ts_tablet_manager.cc:1367] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.009s
I20240502 08:18:11.452795 16332 raft_consensus.cc:348] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.453130 16332 raft_consensus.cc:374] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.453248 16332 raft_consensus.cc:729] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.453569 16332 consensus_queue.cc:260] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.455085 16332 ts_tablet_manager.cc:1398] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:11.455462 16332 tablet_bootstrap.cc:492] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:11.459091 16332 tablet_bootstrap.cc:654] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.460580 16425 tablet.cc:1722] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:11.468412 16440 tablet_bootstrap.cc:492] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:11.468652 16440 ts_tablet_manager.cc:1367] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20240502 08:18:11.469614 16440 raft_consensus.cc:348] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.469913 16440 raft_consensus.cc:374] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.470103 16440 raft_consensus.cc:729] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.470388 16440 consensus_queue.cc:260] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.471390 16438 tablet_bootstrap.cc:492] T 42e0e206eca841fd8dfa7146f8f2d890 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:11.471544 16438 ts_tablet_manager.cc:1367] T 42e0e206eca841fd8dfa7146f8f2d890 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.029s
I20240502 08:18:11.471763 16442 tablet_bootstrap.cc:492] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:11.471910 16442 ts_tablet_manager.cc:1367] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.053s user 0.021s sys 0.021s
I20240502 08:18:11.472294 16438 raft_consensus.cc:348] T 42e0e206eca841fd8dfa7146f8f2d890 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.472894 16438 raft_consensus.cc:374] T 42e0e206eca841fd8dfa7146f8f2d890 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.473011 16438 raft_consensus.cc:729] T 42e0e206eca841fd8dfa7146f8f2d890 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.473354 16332 tablet_bootstrap.cc:492] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:11.473263 16438 consensus_queue.cc:260] T 42e0e206eca841fd8dfa7146f8f2d890 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.473498 16332 ts_tablet_manager.cc:1367] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240502 08:18:11.474298 16332 raft_consensus.cc:348] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.474509 16332 raft_consensus.cc:374] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.474599 16332 raft_consensus.cc:729] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.472584 16442 raft_consensus.cc:348] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.474807 16442 raft_consensus.cc:374] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.474911 16442 raft_consensus.cc:729] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.475138 16442 consensus_queue.cc:260] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.475669 16442 ts_tablet_manager.cc:1398] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:18:11.475952 16332 consensus_queue.cc:260] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.477593 16332 ts_tablet_manager.cc:1398] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:11.477870 16332 tablet_bootstrap.cc:492] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:11.479099 16441 tablet_bootstrap.cc:492] T 42e0e206eca841fd8dfa7146f8f2d890 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:11.479257 16441 ts_tablet_manager.cc:1367] T 42e0e206eca841fd8dfa7146f8f2d890 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.003s
I20240502 08:18:11.479349 16438 ts_tablet_manager.cc:1398] T 42e0e206eca841fd8dfa7146f8f2d890 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 08:18:11.480253 16332 tablet_bootstrap.cc:654] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.480759 16441 raft_consensus.cc:348] T 42e0e206eca841fd8dfa7146f8f2d890 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.480962 16441 raft_consensus.cc:374] T 42e0e206eca841fd8dfa7146f8f2d890 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.481048 16441 raft_consensus.cc:729] T 42e0e206eca841fd8dfa7146f8f2d890 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.481281 16441 consensus_queue.cc:260] T 42e0e206eca841fd8dfa7146f8f2d890 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.483305 16441 ts_tablet_manager.cc:1398] T 42e0e206eca841fd8dfa7146f8f2d890 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:11.483565 16441 tablet_bootstrap.cc:492] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:11.485180 16441 tablet_bootstrap.cc:654] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.485738 16440 ts_tablet_manager.cc:1398] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 08:18:11.490554 16441 tablet_bootstrap.cc:492] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:11.490682 16441 ts_tablet_manager.cc:1367] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240502 08:18:11.491168 16441 raft_consensus.cc:348] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.491312 16441 raft_consensus.cc:374] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.491364 16441 raft_consensus.cc:729] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.491545 16441 consensus_queue.cc:260] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.491981 16441 ts_tablet_manager.cc:1398] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:18:11.492301 16441 tablet_bootstrap.cc:492] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:11.493822 16441 tablet_bootstrap.cc:654] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.495760 16441 tablet_bootstrap.cc:492] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:11.495875 16441 ts_tablet_manager.cc:1367] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:11.496450 16441 raft_consensus.cc:348] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.496596 16441 raft_consensus.cc:374] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.496697 16441 raft_consensus.cc:729] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.496958 16441 consensus_queue.cc:260] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.497400 16441 ts_tablet_manager.cc:1398] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:18:11.506852 16332 tablet_bootstrap.cc:492] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:11.507010 16332 ts_tablet_manager.cc:1367] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.009s
I20240502 08:18:11.507582 16332 raft_consensus.cc:348] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.507755 16332 raft_consensus.cc:374] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.507820 16332 raft_consensus.cc:729] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.508056 16332 consensus_queue.cc:260] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.508469 16332 ts_tablet_manager.cc:1398] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:18:11.537309 16293 raft_consensus.cc:482] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.537457 16293 raft_consensus.cc:504] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.538024 16293 leader_election.cc:288] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:11.538395 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8774c037b901461684d96460e663d17e" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:11.538528 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8774c037b901461684d96460e663d17e" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:11.538619 15886 raft_consensus.cc:2454] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:11.538730 15487 raft_consensus.cc:2454] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:11.539047 15555 leader_election.cc:302] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:11.539307 16293 raft_consensus.cc:2781] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.539386 16293 raft_consensus.cc:482] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.539479 16293 raft_consensus.cc:3037] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.541189 16293 raft_consensus.cc:504] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.541671 16293 leader_election.cc:288] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:11.542053 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8774c037b901461684d96460e663d17e" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:11.542052 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8774c037b901461684d96460e663d17e" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:11.542212 15487 raft_consensus.cc:3037] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.542213 15886 raft_consensus.cc:3037] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.544850 15487 raft_consensus.cc:2454] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:11.544843 15886 raft_consensus.cc:2454] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:11.545238 15555 leader_election.cc:302] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:11.545488 16293 raft_consensus.cc:2781] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.545869 16293 raft_consensus.cc:686] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:11.546129 16293 consensus_queue.cc:237] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.549126 12579 catalog_manager.cc:5484] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:11.553254 12579 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:11.553449 12579 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:11.555881 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 421829e63d274a978e337607efb2cff3
I20240502 08:18:11.556615 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet 421829e63d274a978e337607efb2cff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:11 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:11.556660 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet 421829e63d274a978e337607efb2cff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:11 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:11.556874 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 421829e63d274a978e337607efb2cff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:11 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:11.557250 16453 tablet_replica.cc:330] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:11.557493 16454 tablet_replica.cc:330] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:11.557495 16453 raft_consensus.cc:2229] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:11.557665 16455 tablet_replica.cc:330] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:11.557716 16453 raft_consensus.cc:2258] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:11.557713 16454 raft_consensus.cc:2229] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:11.557880 16455 raft_consensus.cc:2229] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:11.558108 16455 raft_consensus.cc:2258] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:11.558568 16453 ts_tablet_manager.cc:1875] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:11.558056 16454 raft_consensus.cc:2258] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:11.558975 16455 ts_tablet_manager.cc:1875] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:11.559839 16454 ts_tablet_manager.cc:1875] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:11.560124 16373 raft_consensus.cc:482] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.560285 16373 raft_consensus.cc:504] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.561048 16373 leader_election.cc:288] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:11.561772 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e0e206eca841fd8dfa7146f8f2d890" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:11.562021 15755 raft_consensus.cc:2454] T 42e0e206eca841fd8dfa7146f8f2d890 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:11.562821 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e0e206eca841fd8dfa7146f8f2d890" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:11.563035 15622 raft_consensus.cc:2454] T 42e0e206eca841fd8dfa7146f8f2d890 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:11.563510 15418 leader_election.cc:302] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:11.564848 16453 ts_tablet_manager.cc:1888] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:11.565001 16453 log.cc:1196] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/421829e63d274a978e337607efb2cff3
I20240502 08:18:11.565266 16455 ts_tablet_manager.cc:1888] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:11.565412 16455 log.cc:1196] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/421829e63d274a978e337607efb2cff3
I20240502 08:18:11.565464 16453 ts_tablet_manager.cc:1909] T 421829e63d274a978e337607efb2cff3 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:11.565826 16455 ts_tablet_manager.cc:1909] T 421829e63d274a978e337607efb2cff3 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:11.565852 16373 raft_consensus.cc:2781] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.565984 16373 raft_consensus.cc:482] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.566089 16373 raft_consensus.cc:3037] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.567288 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 421829e63d274a978e337607efb2cff3 (table rebalancer_dynamic_table_new_name [id=de593c2fc465447cb6c70e3db40a5e18]) successfully deleted
I20240502 08:18:11.567518 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 421829e63d274a978e337607efb2cff3 (table rebalancer_dynamic_table_new_name [id=de593c2fc465447cb6c70e3db40a5e18]) successfully deleted
I20240502 08:18:11.568610 16373 raft_consensus.cc:504] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.568939 16454 ts_tablet_manager.cc:1888] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:11.569074 16454 log.cc:1196] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/421829e63d274a978e337607efb2cff3
I20240502 08:18:11.569555 16454 ts_tablet_manager.cc:1909] T 421829e63d274a978e337607efb2cff3 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:11.569731 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e0e206eca841fd8dfa7146f8f2d890" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:11.569902 15622 raft_consensus.cc:3037] T 42e0e206eca841fd8dfa7146f8f2d890 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.569933 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e0e206eca841fd8dfa7146f8f2d890" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:11.570115 15755 raft_consensus.cc:3037] T 42e0e206eca841fd8dfa7146f8f2d890 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.570974 16373 leader_election.cc:288] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:11.571286 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 421829e63d274a978e337607efb2cff3 (table rebalancer_dynamic_table_new_name [id=de593c2fc465447cb6c70e3db40a5e18]) successfully deleted
I20240502 08:18:11.572047 15622 raft_consensus.cc:2454] T 42e0e206eca841fd8dfa7146f8f2d890 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:11.572382 15755 raft_consensus.cc:2454] T 42e0e206eca841fd8dfa7146f8f2d890 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:11.572412 15421 leader_election.cc:302] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:11.572695 16373 raft_consensus.cc:2781] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.572844 16373 raft_consensus.cc:686] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:11.573135 16373 consensus_queue.cc:237] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.575075 16373 raft_consensus.cc:482] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.575206 16373 raft_consensus.cc:504] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.575824 16373 leader_election.cc:288] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:11.575887 12579 catalog_manager.cc:5484] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:11.576221 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5205c45461da48369a6f0d103f8a94e6" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:11.576432 15622 raft_consensus.cc:2454] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:11.576795 15421 leader_election.cc:302] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:11.577075 16373 raft_consensus.cc:2781] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.577164 16373 raft_consensus.cc:482] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.577275 16373 raft_consensus.cc:3037] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.579306 16373 raft_consensus.cc:504] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.579897 16373 leader_election.cc:288] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:11.580277 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5205c45461da48369a6f0d103f8a94e6" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:11.580435 15622 raft_consensus.cc:3037] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.582311 15622 raft_consensus.cc:2454] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:11.582754 15421 leader_election.cc:302] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:11.582986 16373 raft_consensus.cc:2781] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.583122 16373 raft_consensus.cc:686] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:11.583364 16373 consensus_queue.cc:237] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:11.585893 12579 catalog_manager.cc:5484] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:11.593195 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5205c45461da48369a6f0d103f8a94e6" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:11.593228 12991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5205c45461da48369a6f0d103f8a94e6" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:11.593464 12992 raft_consensus.cc:2454] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:11.657876 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:11.664817 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 30f1ec355901462cafef8aa15c3fec57 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75d1e74590ff42a49548739f7baca9e5]), partition=
I20240502 08:18:11.665122 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 30f1ec355901462cafef8aa15c3fec57 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75d1e74590ff42a49548739f7baca9e5]), partition=
I20240502 08:18:11.665220 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f1ec355901462cafef8aa15c3fec57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.665429 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f1ec355901462cafef8aa15c3fec57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.667321 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 30f1ec355901462cafef8aa15c3fec57 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75d1e74590ff42a49548739f7baca9e5]), partition=
I20240502 08:18:11.667676 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f1ec355901462cafef8aa15c3fec57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:11.671223 16332 tablet_bootstrap.cc:492] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:11.672572 16440 tablet_bootstrap.cc:492] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:11.672688 16332 tablet_bootstrap.cc:654] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.674192 16440 tablet_bootstrap.cc:654] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.675288 16438 tablet_bootstrap.cc:492] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:11.676157 16440 tablet_bootstrap.cc:492] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:11.676322 16440 ts_tablet_manager.cc:1367] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 08:18:11.676716 16438 tablet_bootstrap.cc:654] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:11.677134 16440 raft_consensus.cc:348] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:11.677310 16440 raft_consensus.cc:374] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.677413 16440 raft_consensus.cc:729] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.677659 16440 consensus_queue.cc:260] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:11.678854 16440 ts_tablet_manager.cc:1398] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:18:11.679834 16438 tablet_bootstrap.cc:492] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:11.679987 16438 ts_tablet_manager.cc:1367] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:18:11.680676 16438 raft_consensus.cc:348] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:11.680964 16438 raft_consensus.cc:374] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.681115 16438 raft_consensus.cc:729] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.681387 16438 consensus_queue.cc:260] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:11.682405 16438 ts_tablet_manager.cc:1398] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:11.683459 16332 tablet_bootstrap.cc:492] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:11.683605 16332 ts_tablet_manager.cc:1367] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240502 08:18:11.684275 16332 raft_consensus.cc:348] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:11.684499 16332 raft_consensus.cc:374] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:11.684600 16332 raft_consensus.cc:729] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:11.684825 16332 consensus_queue.cc:260] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:11.685489 16332 ts_tablet_manager.cc:1398] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:11.829696 16228 raft_consensus.cc:482] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:11.829855 16228 raft_consensus.cc:504] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:11.830603 16228 leader_election.cc:288] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:11.831040 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f1ec355901462cafef8aa15c3fec57" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:11.831066 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f1ec355901462cafef8aa15c3fec57" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:11.831266 15886 raft_consensus.cc:2454] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:11.831287 15487 raft_consensus.cc:2454] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:11.831604 15688 leader_election.cc:302] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:11.831842 16228 raft_consensus.cc:2781] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:11.831946 16228 raft_consensus.cc:482] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:11.832032 16228 raft_consensus.cc:3037] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.833781 16228 raft_consensus.cc:504] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:11.834252 16228 leader_election.cc:288] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:11.834607 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f1ec355901462cafef8aa15c3fec57" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:11.834666 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f1ec355901462cafef8aa15c3fec57" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:11.834811 15886 raft_consensus.cc:3037] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.834857 15487 raft_consensus.cc:3037] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:11.837384 15886 raft_consensus.cc:2454] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:11.837394 15487 raft_consensus.cc:2454] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:11.837726 15688 leader_election.cc:302] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:11.837965 16228 raft_consensus.cc:2781] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:11.838325 16228 raft_consensus.cc:686] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:11.838753 16228 consensus_queue.cc:237] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:11.841413 12579 catalog_manager.cc:5484] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:11.949168 16293 consensus_queue.cc:1035] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.959993 16451 consensus_queue.cc:1035] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.962903 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:11.969707 15488 raft_consensus.cc:1261] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.970022 15886 raft_consensus.cc:1261] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:11.970661 16228 consensus_queue.cc:1035] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.971336 16464 consensus_queue.cc:1035] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.975517 16470 mvcc.cc:205] Tried to move back new op lower bound from 7023156805500080128 to 7023156804973481984. Current Snapshot: MvccSnapshot[applied={T|T < 7023156805500080128}]
I20240502 08:18:11.975785 16473 tablet.cc:1722] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:11.976366 16474 tablet.cc:1722] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:11.977046 16474 tablet.cc:1620] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:11.978817 16473 tablet.cc:1620] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:11.984426 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=75d1e74590ff42a49548739f7baca9e5] alter complete (version 1)
I20240502 08:18:11.988368 12584 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 30f1ec355901462cafef8aa15c3fec57 (table rebalancer_dynamic_table [id=75d1e74590ff42a49548739f7baca9e5]). Expected version 1 got 0
I20240502 08:18:11.988389 16476 tablet.cc:1722] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:11.989033 16476 tablet.cc:1620] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:12.087450 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:12.094512 16473 tablet.cc:1722] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:12.095052 16473 tablet.cc:1620] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:12.095224 16476 tablet.cc:1722] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:12.095517 16474 tablet.cc:1722] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:12.095947 16474 tablet.cc:1620] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:12.096529 16476 tablet.cc:1620] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:12.104308 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=75d1e74590ff42a49548739f7baca9e5] alter complete (version 2)
I20240502 08:18:12.105935 16373 consensus_queue.cc:1035] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.107455 16478 consensus_queue.cc:1035] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.111194 16478 consensus_queue.cc:1035] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.115985 12587 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:12.124197 12992 raft_consensus.cc:3037] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.125324 12840 tablet_service.cc:1467] Processing CreateTablet for tablet 8c28e9d3d29548a894b520ae7b324129 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=70609143f5404be1badc9b37bce65793]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.125689 12840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c28e9d3d29548a894b520ae7b324129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.126794 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 8c28e9d3d29548a894b520ae7b324129 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=70609143f5404be1badc9b37bce65793]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.127182 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c28e9d3d29548a894b520ae7b324129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.127890 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 0d57fc923fe946afa65a6aa8ce52ba7b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=70609143f5404be1badc9b37bce65793]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.128466 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 0d57fc923fe946afa65a6aa8ce52ba7b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=70609143f5404be1badc9b37bce65793]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.128485 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 8c28e9d3d29548a894b520ae7b324129 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=70609143f5404be1badc9b37bce65793]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.128868 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d57fc923fe946afa65a6aa8ce52ba7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.128895 15467 tablet_service.cc:1467] Processing CreateTablet for tablet b5cd9e39bfc740b78492d369532d0c66 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=70609143f5404be1badc9b37bce65793]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.129247 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cd9e39bfc740b78492d369532d0c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.129484 12842 tablet_service.cc:1467] Processing CreateTablet for tablet b5cd9e39bfc740b78492d369532d0c66 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=70609143f5404be1badc9b37bce65793]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.129698 15735 tablet_service.cc:1467] Processing CreateTablet for tablet b5cd9e39bfc740b78492d369532d0c66 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=70609143f5404be1badc9b37bce65793]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.129880 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cd9e39bfc740b78492d369532d0c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.130070 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cd9e39bfc740b78492d369532d0c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.129503 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 0d57fc923fe946afa65a6aa8ce52ba7b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=70609143f5404be1badc9b37bce65793]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.132668 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d57fc923fe946afa65a6aa8ce52ba7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.134662 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d57fc923fe946afa65a6aa8ce52ba7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.135788 16332 tablet_bootstrap.cc:492] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:12.137130 16478 consensus_queue.cc:1035] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.137346 16332 tablet_bootstrap.cc:654] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.138646 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c28e9d3d29548a894b520ae7b324129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.148905 16438 tablet_bootstrap.cc:492] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:12.150400 16438 tablet_bootstrap.cc:654] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.151993 16485 tablet_bootstrap.cc:492] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:12.153075 16486 tablet_bootstrap.cc:492] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:12.153134 16440 tablet_bootstrap.cc:492] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:12.154578 16485 tablet_bootstrap.cc:654] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.154767 16486 tablet_bootstrap.cc:654] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.157382 16440 tablet_bootstrap.cc:654] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.160751 16438 tablet_bootstrap.cc:492] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:12.161767 16332 tablet_bootstrap.cc:492] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:12.162014 16332 ts_tablet_manager.cc:1367] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.007s
I20240502 08:18:12.162883 16332 raft_consensus.cc:348] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.163179 16438 ts_tablet_manager.cc:1367] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 08:18:12.163214 16332 raft_consensus.cc:374] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.163290 16332 raft_consensus.cc:729] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.163688 16332 consensus_queue.cc:260] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.165695 16332 ts_tablet_manager.cc:1398] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:12.166069 16438 raft_consensus.cc:348] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.166589 16438 raft_consensus.cc:374] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.167445 16438 raft_consensus.cc:729] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.167773 16438 consensus_queue.cc:260] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.169661 16438 ts_tablet_manager.cc:1398] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:18:12.172909 16485 tablet_bootstrap.cc:492] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:12.173108 16485 ts_tablet_manager.cc:1367] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.010s
I20240502 08:18:12.173904 16485 raft_consensus.cc:348] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.176837 16485 raft_consensus.cc:374] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.177033 16485 raft_consensus.cc:729] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.178292 16485 consensus_queue.cc:260] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.178993 16485 ts_tablet_manager.cc:1398] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 08:18:12.179426 16485 tablet_bootstrap.cc:492] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:12.181047 16485 tablet_bootstrap.cc:654] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.191089 16438 tablet_bootstrap.cc:492] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:12.191705 16485 tablet_bootstrap.cc:492] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:12.191923 16485 ts_tablet_manager.cc:1367] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20240502 08:18:12.192782 16485 raft_consensus.cc:348] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:12.193081 16485 raft_consensus.cc:374] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.193190 16485 raft_consensus.cc:729] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.193529 16485 consensus_queue.cc:260] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:12.194211 16485 ts_tablet_manager.cc:1398] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:12.200495 16438 tablet_bootstrap.cc:654] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.203552 16485 tablet_bootstrap.cc:492] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:12.207708 16485 tablet_bootstrap.cc:654] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.208094 16438 tablet_bootstrap.cc:492] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:12.208335 16438 ts_tablet_manager.cc:1367] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20240502 08:18:12.210145 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:12.213794 16438 raft_consensus.cc:348] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.214268 16438 raft_consensus.cc:374] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.214394 16438 raft_consensus.cc:729] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.214751 16438 consensus_queue.cc:260] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.217203 16258 raft_consensus.cc:482] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.217351 16258 raft_consensus.cc:504] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.218106 16258 leader_election.cc:288] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:12.218842 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c28e9d3d29548a894b520ae7b324129" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:12.219061 15755 raft_consensus.cc:2454] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:12.219252 16440 tablet_bootstrap.cc:492] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:12.219409 16440 ts_tablet_manager.cc:1367] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.007s
I20240502 08:18:12.219990 12795 leader_election.cc:302] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:12.220227 16440 raft_consensus.cc:348] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.220346 16258 raft_consensus.cc:2781] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:12.220664 16440 raft_consensus.cc:374] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.220715 16258 raft_consensus.cc:482] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:12.220829 16258 raft_consensus.cc:3037] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.220861 16440 raft_consensus.cc:729] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.221175 16440 consensus_queue.cc:260] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.223685 16258 raft_consensus.cc:504] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.224900 16440 ts_tablet_manager.cc:1398] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240502 08:18:12.225214 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c28e9d3d29548a894b520ae7b324129" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:12.225405 15755 raft_consensus.cc:3037] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.227797 16440 tablet_bootstrap.cc:492] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:12.228024 15755 raft_consensus.cc:2454] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:12.228868 12795 leader_election.cc:302] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:12.229941 16440 tablet_bootstrap.cc:654] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.232474 16258 leader_election.cc:288] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:12.232672 16258 raft_consensus.cc:2781] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:12.232800 16258 raft_consensus.cc:686] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:18:12.232985 16473 tablet.cc:1722] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:12.233052 16258 consensus_queue.cc:237] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.235611 16440 tablet_bootstrap.cc:492] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:12.235764 16440 ts_tablet_manager.cc:1367] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240502 08:18:12.238934 16474 tablet.cc:1722] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:12.242960 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=75d1e74590ff42a49548739f7baca9e5] alter complete (version 3)
I20240502 08:18:12.243649 16438 ts_tablet_manager.cc:1398] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240502 08:18:12.245903 16440 raft_consensus.cc:348] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:12.246090 16440 raft_consensus.cc:374] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.246171 16440 raft_consensus.cc:729] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.246403 16440 consensus_queue.cc:260] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:12.247133 16440 ts_tablet_manager.cc:1398] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 08:18:12.252177 16486 tablet_bootstrap.cc:492] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:12.252342 16486 ts_tablet_manager.cc:1367] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.099s user 0.024s sys 0.032s
I20240502 08:18:12.253249 16373 raft_consensus.cc:482] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.253417 16373 raft_consensus.cc:504] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.253495 16486 raft_consensus.cc:348] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:12.253711 16486 raft_consensus.cc:374] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.253805 16486 raft_consensus.cc:729] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.254017 16373 leader_election.cc:288] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:12.254016 16486 consensus_queue.cc:260] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:12.255458 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cd9e39bfc740b78492d369532d0c66" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
W20240502 08:18:12.256582 15419 leader_election.cc:341] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:12.260088 16486 ts_tablet_manager.cc:1398] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 08:18:12.260098 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cd9e39bfc740b78492d369532d0c66" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:12.261286 16485 tablet_bootstrap.cc:492] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:12.261438 16485 ts_tablet_manager.cc:1367] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.011s
I20240502 08:18:12.262202 16485 raft_consensus.cc:348] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.262400 16485 raft_consensus.cc:374] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.262506 16485 raft_consensus.cc:729] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.260427 15755 raft_consensus.cc:2454] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:12.262782 16485 consensus_queue.cc:260] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.263288 16485 ts_tablet_manager.cc:1398] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:12.265451 15418 leader_election.cc:302] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters: e1dd14f7ca764a1490df419f13f2804a
I20240502 08:18:12.265964 16373 raft_consensus.cc:2781] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:12.266049 16373 raft_consensus.cc:482] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:12.266129 16373 raft_consensus.cc:3037] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.268934 16373 raft_consensus.cc:504] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.269899 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cd9e39bfc740b78492d369532d0c66" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:12.270130 12859 raft_consensus.cc:3037] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.272817 12859 raft_consensus.cc:2454] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:12.273238 15419 leader_election.cc:302] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db7893bb29a047b3b82cc60bfd8930a3, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:12.273502 16478 raft_consensus.cc:2781] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:12.273641 16478 raft_consensus.cc:686] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:12.273900 16478 consensus_queue.cc:237] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:12.275192 16476 tablet.cc:1722] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:12.277079 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cd9e39bfc740b78492d369532d0c66" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:12.277326 15755 raft_consensus.cc:3037] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.277521 12579 catalog_manager.cc:5484] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:12.281594 16373 leader_election.cc:288] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:12.283345 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c28e9d3d29548a894b520ae7b324129" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:12.283578 15886 raft_consensus.cc:2454] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:12.283877 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c28e9d3d29548a894b520ae7b324129" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:12.284083 15887 raft_consensus.cc:3037] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.285188 12584 catalog_manager.cc:5484] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:12.285707 15755 raft_consensus.cc:2454] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:12.286578 15887 raft_consensus.cc:2454] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:12.346994 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:12.347263 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:12.350314 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 30f1ec355901462cafef8aa15c3fec57
I20240502 08:18:12.351106 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f1ec355901462cafef8aa15c3fec57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:12 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:12.351444 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f1ec355901462cafef8aa15c3fec57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:12 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:12.351461 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f1ec355901462cafef8aa15c3fec57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:12 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:12.351900 16502 tablet_replica.cc:330] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:12.352137 16502 raft_consensus.cc:2229] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:12.352421 16502 raft_consensus.cc:2258] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:12.352838 16504 tablet_replica.cc:330] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:12.353024 16503 tablet_replica.cc:330] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:12.353055 16504 raft_consensus.cc:2229] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:12.353217 16503 raft_consensus.cc:2229] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:12.353304 16504 raft_consensus.cc:2258] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:12.353601 16503 raft_consensus.cc:2258] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:12.354363 16504 ts_tablet_manager.cc:1875] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:12.354737 16503 ts_tablet_manager.cc:1875] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:12.354943 16502 ts_tablet_manager.cc:1875] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:12.360639 16503 ts_tablet_manager.cc:1888] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:12.360767 16502 ts_tablet_manager.cc:1888] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:12.360814 16503 log.cc:1196] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/30f1ec355901462cafef8aa15c3fec57
I20240502 08:18:12.360976 16502 log.cc:1196] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/30f1ec355901462cafef8aa15c3fec57
I20240502 08:18:12.361264 16503 ts_tablet_manager.cc:1909] T 30f1ec355901462cafef8aa15c3fec57 P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:12.361505 16502 ts_tablet_manager.cc:1909] T 30f1ec355901462cafef8aa15c3fec57 P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:12.361701 16504 ts_tablet_manager.cc:1888] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:12.361838 16504 log.cc:1196] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/30f1ec355901462cafef8aa15c3fec57
I20240502 08:18:12.362252 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 30f1ec355901462cafef8aa15c3fec57 (table rebalancer_dynamic_table_new_name [id=75d1e74590ff42a49548739f7baca9e5]) successfully deleted
I20240502 08:18:12.362316 16504 ts_tablet_manager.cc:1909] T 30f1ec355901462cafef8aa15c3fec57 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:12.362519 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 30f1ec355901462cafef8aa15c3fec57 (table rebalancer_dynamic_table_new_name [id=75d1e74590ff42a49548739f7baca9e5]) successfully deleted
I20240502 08:18:12.363270 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 30f1ec355901462cafef8aa15c3fec57 (table rebalancer_dynamic_table_new_name [id=75d1e74590ff42a49548739f7baca9e5]) successfully deleted
I20240502 08:18:12.452291 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:12.459497 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb2cd7e7a494b32aae22d58ba0d65b6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=796cf932cd884172b86773abdecb78fb]), partition=
I20240502 08:18:12.459766 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb2cd7e7a494b32aae22d58ba0d65b6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=796cf932cd884172b86773abdecb78fb]), partition=
I20240502 08:18:12.459889 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb2cd7e7a494b32aae22d58ba0d65b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.460114 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb2cd7e7a494b32aae22d58ba0d65b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.463016 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb2cd7e7a494b32aae22d58ba0d65b6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=796cf932cd884172b86773abdecb78fb]), partition=
I20240502 08:18:12.463343 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb2cd7e7a494b32aae22d58ba0d65b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.465358 16332 tablet_bootstrap.cc:492] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:12.466760 16332 tablet_bootstrap.cc:654] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.469043 16506 tablet_bootstrap.cc:492] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:12.469130 16507 tablet_bootstrap.cc:492] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:12.469466 16332 tablet_bootstrap.cc:492] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:12.469612 16332 ts_tablet_manager.cc:1367] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:12.470326 16332 raft_consensus.cc:348] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:12.470525 16332 raft_consensus.cc:374] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.470628 16332 raft_consensus.cc:729] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.470736 16506 tablet_bootstrap.cc:654] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.470963 16507 tablet_bootstrap.cc:654] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.471246 16332 consensus_queue.cc:260] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:12.472718 16332 ts_tablet_manager.cc:1398] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:12.474503 16506 tablet_bootstrap.cc:492] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:12.474704 16506 ts_tablet_manager.cc:1367] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:18:12.475487 16506 raft_consensus.cc:348] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:12.475714 16506 raft_consensus.cc:374] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.475802 16506 raft_consensus.cc:729] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.476042 16506 consensus_queue.cc:260] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:12.476729 16506 ts_tablet_manager.cc:1398] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:12.479151 16507 tablet_bootstrap.cc:492] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:12.479492 16507 ts_tablet_manager.cc:1367] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 08:18:12.480257 16507 raft_consensus.cc:348] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:12.480491 16507 raft_consensus.cc:374] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.480592 16507 raft_consensus.cc:729] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.480805 16507 consensus_queue.cc:260] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:12.481351 16507 ts_tablet_manager.cc:1398] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:12.554159 16024 raft_consensus.cc:482] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.554318 16024 raft_consensus.cc:504] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:12.554874 16024 leader_election.cc:288] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:12.555320 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d57fc923fe946afa65a6aa8ce52ba7b" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:12.555367 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d57fc923fe946afa65a6aa8ce52ba7b" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:12.555545 12859 raft_consensus.cc:2454] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:12.555546 12730 raft_consensus.cc:2454] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:12.555897 15819 leader_election.cc:302] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:12.556166 16024 raft_consensus.cc:2781] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:12.556267 16024 raft_consensus.cc:482] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:12.556360 16024 raft_consensus.cc:3037] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.558099 16024 raft_consensus.cc:504] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:12.558668 16024 leader_election.cc:288] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:12.558923 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d57fc923fe946afa65a6aa8ce52ba7b" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:12.559018 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d57fc923fe946afa65a6aa8ce52ba7b" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:12.559094 12730 raft_consensus.cc:3037] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.559192 12859 raft_consensus.cc:3037] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.560922 12730 raft_consensus.cc:2454] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:12.561072 12859 raft_consensus.cc:2454] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:12.561276 15819 leader_election.cc:302] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 29b035434087435cbaf4ebcac98eedee; no voters:
I20240502 08:18:12.561664 16024 raft_consensus.cc:2781] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:12.562036 16024 raft_consensus.cc:686] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:12.562302 16024 consensus_queue.cc:237] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:12.565025 12587 catalog_manager.cc:5484] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:12.687743 16400 raft_consensus.cc:482] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.687885 16400 raft_consensus.cc:504] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:12.688578 16400 leader_election.cc:288] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:12.708196 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb2cd7e7a494b32aae22d58ba0d65b6" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:12.708437 15487 raft_consensus.cc:2454] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:12.708911 12926 leader_election.cc:302] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:12.709203 16400 raft_consensus.cc:2781] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:12.709313 16400 raft_consensus.cc:482] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:12.709412 16400 raft_consensus.cc:3037] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.709931 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb2cd7e7a494b32aae22d58ba0d65b6" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:12.710129 15622 raft_consensus.cc:2454] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:12.712036 16400 raft_consensus.cc:504] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:12.712679 16400 leader_election.cc:288] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:12.713232 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb2cd7e7a494b32aae22d58ba0d65b6" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:12.713261 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb2cd7e7a494b32aae22d58ba0d65b6" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:12.713411 15622 raft_consensus.cc:3037] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.713438 15487 raft_consensus.cc:3037] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.715411 15622 raft_consensus.cc:2454] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:12.715425 15487 raft_consensus.cc:2454] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:12.715765 12928 leader_election.cc:302] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:12.716045 16400 raft_consensus.cc:2781] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:12.716429 16400 raft_consensus.cc:686] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:12.716696 16400 consensus_queue.cc:237] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:12.719612 12587 catalog_manager.cc:5484] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:12.740085 16373 consensus_queue.cc:1035] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.744426 16373 consensus_queue.cc:1035] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.792481 16497 consensus_queue.cc:1035] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.796324 16526 consensus_queue.cc:1035] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.856640 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:12.861878 15622 raft_consensus.cc:1261] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:12.861882 15487 raft_consensus.cc:1261] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:12.862390 16517 consensus_queue.cc:1035] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.862941 16400 consensus_queue.cc:1035] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.867970 12587 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet 9fb2cd7e7a494b32aae22d58ba0d65b6 (table rebalancer_dynamic_table [id=796cf932cd884172b86773abdecb78fb]). Expected version 1 got 0
I20240502 08:18:12.869812 16533 tablet.cc:1722] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:12.870446 16533 tablet.cc:1620] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:12.870479 16534 tablet.cc:1722] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:12.870967 16534 tablet.cc:1620] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:12.873677 12587 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet 9fb2cd7e7a494b32aae22d58ba0d65b6 (table rebalancer_dynamic_table [id=796cf932cd884172b86773abdecb78fb]). Expected version 1 got 0
I20240502 08:18:12.877525 16535 tablet.cc:1722] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:12.877987 16535 tablet.cc:1620] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:12.879544 12583 catalog_manager.cc:5576] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983) does not have the latest schema for tablet 9fb2cd7e7a494b32aae22d58ba0d65b6 (table rebalancer_dynamic_table [id=796cf932cd884172b86773abdecb78fb]). Expected version 1 got 0
I20240502 08:18:12.880399 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=796cf932cd884172b86773abdecb78fb] alter complete (version 1)
I20240502 08:18:12.983347 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:12.986089 16511 consensus_queue.cc:1035] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.993956 16534 tablet.cc:1722] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:12.994062 16511 consensus_queue.cc:1035] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:12.994531 16534 tablet.cc:1620] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:12.995054 16535 tablet.cc:1722] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:12.995527 16535 tablet.cc:1620] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:12.996209 16533 tablet.cc:1722] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:12.996685 16533 tablet.cc:1620] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:13.003448 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=796cf932cd884172b86773abdecb78fb] alter complete (version 2)
I20240502 08:18:13.070003 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:13.078548 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 96ab6c62a13447c38e1ec48870a822c3 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ac73d0460aa5473a8d5bdc70b924fe3f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.079021 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ab6c62a13447c38e1ec48870a822c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.079409 15600 tablet_service.cc:1467] Processing CreateTablet for tablet 432a6e076bf4401bb81903d91de38a57 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ac73d0460aa5473a8d5bdc70b924fe3f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.079826 15600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432a6e076bf4401bb81903d91de38a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.080065 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 2e76ea85438940399e164313caddf121 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ac73d0460aa5473a8d5bdc70b924fe3f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.080404 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e76ea85438940399e164313caddf121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.082784 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 432a6e076bf4401bb81903d91de38a57 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ac73d0460aa5473a8d5bdc70b924fe3f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.083120 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432a6e076bf4401bb81903d91de38a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.086380 16540 tablet_bootstrap.cc:492] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:13.086835 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 432a6e076bf4401bb81903d91de38a57 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ac73d0460aa5473a8d5bdc70b924fe3f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.087064 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 2e76ea85438940399e164313caddf121 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ac73d0460aa5473a8d5bdc70b924fe3f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.087251 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432a6e076bf4401bb81903d91de38a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.087417 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e76ea85438940399e164313caddf121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.087922 16540 tablet_bootstrap.cc:654] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.091368 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 96ab6c62a13447c38e1ec48870a822c3 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ac73d0460aa5473a8d5bdc70b924fe3f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.091390 12710 tablet_service.cc:1467] Processing CreateTablet for tablet 2e76ea85438940399e164313caddf121 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ac73d0460aa5473a8d5bdc70b924fe3f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.091820 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ab6c62a13447c38e1ec48870a822c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.091727 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 96ab6c62a13447c38e1ec48870a822c3 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=ac73d0460aa5473a8d5bdc70b924fe3f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.092015 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e76ea85438940399e164313caddf121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.092165 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ab6c62a13447c38e1ec48870a822c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.098596 16542 tablet_bootstrap.cc:492] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:13.100201 16542 tablet_bootstrap.cc:654] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.102012 16544 tablet_bootstrap.cc:492] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:13.103529 16540 tablet_bootstrap.cc:492] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:13.103713 16540 ts_tablet_manager.cc:1367] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20240502 08:18:13.104245 16544 tablet_bootstrap.cc:654] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.104499 16540 raft_consensus.cc:348] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.104753 16540 raft_consensus.cc:374] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.104894 16540 raft_consensus.cc:729] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.105151 16540 consensus_queue.cc:260] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.105728 16540 ts_tablet_manager.cc:1398] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:13.106108 16540 tablet_bootstrap.cc:492] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:13.107559 16540 tablet_bootstrap.cc:654] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.108989 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:13.115512 16542 tablet_bootstrap.cc:492] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:13.115686 16542 ts_tablet_manager.cc:1367] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20240502 08:18:13.117084 16545 tablet_bootstrap.cc:492] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:13.116711 16542 raft_consensus.cc:348] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.117193 16542 raft_consensus.cc:374] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.117303 16542 raft_consensus.cc:729] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.117525 16533 tablet.cc:1722] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:13.118854 16543 tablet_bootstrap.cc:492] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:13.119243 16534 tablet.cc:1722] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:13.120378 16543 tablet_bootstrap.cc:654] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.122009 16545 tablet_bootstrap.cc:654] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.125854 16543 tablet_bootstrap.cc:492] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:13.125999 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=796cf932cd884172b86773abdecb78fb] alter complete (version 3)
I20240502 08:18:13.126049 16543 ts_tablet_manager.cc:1367] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 08:18:13.126917 16543 raft_consensus.cc:348] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:13.127200 16543 raft_consensus.cc:374] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.127323 16543 raft_consensus.cc:729] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.127657 16540 tablet_bootstrap.cc:492] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:13.127816 16540 ts_tablet_manager.cc:1367] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240502 08:18:13.127657 16543 consensus_queue.cc:260] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:13.128464 16543 ts_tablet_manager.cc:1398] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:13.128697 16540 raft_consensus.cc:348] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.128940 16540 raft_consensus.cc:374] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.129029 16540 raft_consensus.cc:729] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.129248 16540 consensus_queue.cc:260] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.129853 16540 ts_tablet_manager.cc:1398] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:13.130132 16540 tablet_bootstrap.cc:492] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:13.131089 16544 tablet_bootstrap.cc:492] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:13.131242 16544 ts_tablet_manager.cc:1367] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240502 08:18:13.131493 16540 tablet_bootstrap.cc:654] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.131955 16544 raft_consensus.cc:348] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.132467 16542 consensus_queue.cc:260] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.132802 16241 raft_consensus.cc:482] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:13.132937 16544 raft_consensus.cc:374] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.132948 16241 raft_consensus.cc:504] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.133040 16544 raft_consensus.cc:729] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.133275 16544 consensus_queue.cc:260] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.133857 16241 leader_election.cc:288] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:13.134299 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e76ea85438940399e164313caddf121" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:13.134511 15487 raft_consensus.cc:2454] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:13.134979 12664 leader_election.cc:302] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:13.136588 16501 raft_consensus.cc:2781] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:13.136683 16501 raft_consensus.cc:482] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:13.136794 16501 raft_consensus.cc:3037] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.139436 16501 raft_consensus.cc:504] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.140040 16501 leader_election.cc:288] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:13.140750 16544 ts_tablet_manager.cc:1398] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:18:13.140985 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e76ea85438940399e164313caddf121" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:13.141187 15487 raft_consensus.cc:3037] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.144187 16544 tablet_bootstrap.cc:492] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:13.145494 15487 raft_consensus.cc:2454] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:13.146395 12664 leader_election.cc:302] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:13.146752 16241 raft_consensus.cc:2781] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:13.146908 16241 raft_consensus.cc:686] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:18:13.147205 16544 tablet_bootstrap.cc:654] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.147159 16241 consensus_queue.cc:237] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.148248 16535 tablet.cc:1722] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:13.159922 12584 catalog_manager.cc:5484] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:13.161144 16542 ts_tablet_manager.cc:1398] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.045s user 0.000s sys 0.002s
I20240502 08:18:13.161466 16542 tablet_bootstrap.cc:492] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:13.163456 16542 tablet_bootstrap.cc:654] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.165926 16545 tablet_bootstrap.cc:492] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:13.167201 16545 ts_tablet_manager.cc:1367] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.003s
I20240502 08:18:13.169175 16451 raft_consensus.cc:482] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:13.169366 16451 raft_consensus.cc:504] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.170135 16451 leader_election.cc:288] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:13.170826 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432a6e076bf4401bb81903d91de38a57" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
W20240502 08:18:13.173243 15554 leader_election.cc:341] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:13.175437 16545 raft_consensus.cc:348] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.175786 16545 raft_consensus.cc:374] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.175916 16545 raft_consensus.cc:729] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.176192 16545 consensus_queue.cc:260] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.180822 16545 ts_tablet_manager.cc:1398] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240502 08:18:13.185667 16544 tablet_bootstrap.cc:492] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:13.185895 16544 ts_tablet_manager.cc:1367] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.017s
I20240502 08:18:13.186844 16544 raft_consensus.cc:348] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.187469 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e76ea85438940399e164313caddf121" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:13.187548 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e76ea85438940399e164313caddf121" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:13.187687 15622 raft_consensus.cc:2454] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:13.188356 16540 tablet_bootstrap.cc:492] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:13.188525 16544 raft_consensus.cc:374] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.188549 16540 ts_tablet_manager.cc:1367] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.059s user 0.022s sys 0.033s
I20240502 08:18:13.188622 16544 raft_consensus.cc:729] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.188899 16544 consensus_queue.cc:260] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.189250 16540 raft_consensus.cc:348] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:13.189435 16540 raft_consensus.cc:374] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.189532 16540 raft_consensus.cc:729] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.189725 16540 consensus_queue.cc:260] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:13.190354 16540 ts_tablet_manager.cc:1398] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:13.190407 16544 ts_tablet_manager.cc:1398] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:13.191223 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432a6e076bf4401bb81903d91de38a57" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:13.191433 13119 raft_consensus.cc:2454] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:13.191870 15553 leader_election.cc:302] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:13.192210 16451 raft_consensus.cc:2781] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:13.192307 16451 raft_consensus.cc:482] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:13.192433 16451 raft_consensus.cc:3037] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.193457 16542 tablet_bootstrap.cc:492] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:13.193682 16542 ts_tablet_manager.cc:1367] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.004s
I20240502 08:18:13.194547 16542 raft_consensus.cc:348] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:13.194814 16542 raft_consensus.cc:374] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.194941 16542 raft_consensus.cc:729] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.195287 16542 consensus_queue.cc:260] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:13.195878 16542 ts_tablet_manager.cc:1398] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:13.197527 16451 raft_consensus.cc:504] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.198391 16451 leader_election.cc:288] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:13.198647 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432a6e076bf4401bb81903d91de38a57" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:13.198822 13119 raft_consensus.cc:3037] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.199174 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432a6e076bf4401bb81903d91de38a57" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:13.199362 15487 raft_consensus.cc:3037] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.201292 13119 raft_consensus.cc:2454] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:13.201680 15553 leader_election.cc:302] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:13.201949 16451 raft_consensus.cc:2781] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:13.201995 15487 raft_consensus.cc:2454] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:13.202821 16451 raft_consensus.cc:686] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:13.203099 16451 consensus_queue.cc:237] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:13.205556 12584 catalog_manager.cc:5484] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:13.228896 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:13.229051 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:13.231462 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 9fb2cd7e7a494b32aae22d58ba0d65b6
I20240502 08:18:13.232201 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fb2cd7e7a494b32aae22d58ba0d65b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:13 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:13.232280 15601 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fb2cd7e7a494b32aae22d58ba0d65b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:13 UTC) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:13.232466 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fb2cd7e7a494b32aae22d58ba0d65b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:13 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:13.232887 16562 tablet_replica.cc:330] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:13.233067 16562 raft_consensus.cc:2229] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:13.233280 16562 raft_consensus.cc:2258] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:13.234123 16562 ts_tablet_manager.cc:1875] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:13.239845 16562 ts_tablet_manager.cc:1888] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:13.240028 16562 log.cc:1196] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/9fb2cd7e7a494b32aae22d58ba0d65b6
I20240502 08:18:13.240521 16562 ts_tablet_manager.cc:1909] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting consensus metadata
I20240502 08:18:13.241422 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 9fb2cd7e7a494b32aae22d58ba0d65b6 (table rebalancer_dynamic_table_new_name [id=796cf932cd884172b86773abdecb78fb]) successfully deleted
I20240502 08:18:13.241550 16561 tablet_replica.cc:330] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:13.241634 16563 tablet_replica.cc:330] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:13.241739 16561 raft_consensus.cc:2229] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:13.241938 16561 raft_consensus.cc:2258] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:13.242714 16561 ts_tablet_manager.cc:1875] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:13.242808 16563 raft_consensus.cc:2229] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:13.243100 16563 raft_consensus.cc:2258] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:13.243768 16563 ts_tablet_manager.cc:1875] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:13.247622 16561 ts_tablet_manager.cc:1888] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:13.247716 16561 log.cc:1196] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/9fb2cd7e7a494b32aae22d58ba0d65b6
I20240502 08:18:13.248013 16563 ts_tablet_manager.cc:1888] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:13.248083 16561 ts_tablet_manager.cc:1909] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:13.248118 16563 log.cc:1196] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/9fb2cd7e7a494b32aae22d58ba0d65b6
I20240502 08:18:13.248471 16563 ts_tablet_manager.cc:1909] T 9fb2cd7e7a494b32aae22d58ba0d65b6 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:13.248839 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 9fb2cd7e7a494b32aae22d58ba0d65b6 (table rebalancer_dynamic_table_new_name [id=796cf932cd884172b86773abdecb78fb]) successfully deleted
I20240502 08:18:13.249292 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 9fb2cd7e7a494b32aae22d58ba0d65b6 (table rebalancer_dynamic_table_new_name [id=796cf932cd884172b86773abdecb78fb]) successfully deleted
I20240502 08:18:13.333900 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:13.342427 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 1702f0e988804a9f930900ed4ab266fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=144b219c4b0b46c9a8e3d50f0d424847]), partition=
I20240502 08:18:13.342706 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 1702f0e988804a9f930900ed4ab266fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=144b219c4b0b46c9a8e3d50f0d424847]), partition=
I20240502 08:18:13.342895 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1702f0e988804a9f930900ed4ab266fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.343044 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1702f0e988804a9f930900ed4ab266fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.343103 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 1702f0e988804a9f930900ed4ab266fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=144b219c4b0b46c9a8e3d50f0d424847]), partition=
I20240502 08:18:13.343433 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1702f0e988804a9f930900ed4ab266fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.347487 16543 tablet_bootstrap.cc:492] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:13.349108 16543 tablet_bootstrap.cc:654] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.349130 16540 tablet_bootstrap.cc:492] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:13.350137 16564 tablet_bootstrap.cc:492] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:13.350775 16540 tablet_bootstrap.cc:654] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.351435 16564 tablet_bootstrap.cc:654] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.353492 16501 raft_consensus.cc:482] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:13.353673 16501 raft_consensus.cc:504] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:13.354341 16501 leader_election.cc:288] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:13.355196 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ab6c62a13447c38e1ec48870a822c3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:13.355418 15622 raft_consensus.cc:2454] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:13.355876 12666 leader_election.cc:302] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:13.356410 16501 raft_consensus.cc:2781] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:13.356528 16501 raft_consensus.cc:482] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:13.356647 16501 raft_consensus.cc:3037] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.358446 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ab6c62a13447c38e1ec48870a822c3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:13.358704 15886 raft_consensus.cc:2454] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:13.359603 16501 raft_consensus.cc:504] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:13.360333 16501 leader_election.cc:288] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:13.360922 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ab6c62a13447c38e1ec48870a822c3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:13.361116 15622 raft_consensus.cc:3037] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.362001 16540 tablet_bootstrap.cc:492] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:13.362110 16543 tablet_bootstrap.cc:492] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:13.362223 16540 ts_tablet_manager.cc:1367] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20240502 08:18:13.362498 16543 ts_tablet_manager.cc:1367] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20240502 08:18:13.363113 16540 raft_consensus.cc:348] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.363162 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96ab6c62a13447c38e1ec48870a822c3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:13.363344 16540 raft_consensus.cc:374] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.363354 15886 raft_consensus.cc:3037] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.363437 16540 raft_consensus.cc:729] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.363673 16543 raft_consensus.cc:348] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.363785 15622 raft_consensus.cc:2454] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:13.363734 16540 consensus_queue.cc:260] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.364104 16543 raft_consensus.cc:374] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.364146 12666 leader_election.cc:302] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:13.364243 16543 raft_consensus.cc:729] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.364302 16540 ts_tablet_manager.cc:1398] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:13.364408 16501 raft_consensus.cc:2781] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:13.364537 16501 raft_consensus.cc:686] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:18:13.364768 16501 consensus_queue.cc:237] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:13.365293 16543 consensus_queue.cc:260] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.365900 15886 raft_consensus.cc:2454] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:13.366098 16543 ts_tablet_manager.cc:1398] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:13.369936 16564 tablet_bootstrap.cc:492] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:13.370095 16564 ts_tablet_manager.cc:1367] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.008s
I20240502 08:18:13.369953 12579 catalog_manager.cc:5484] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:13.370821 16564 raft_consensus.cc:348] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.371055 16564 raft_consensus.cc:374] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.371161 16564 raft_consensus.cc:729] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.371435 16564 consensus_queue.cc:260] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.373121 16564 ts_tablet_manager.cc:1398] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:13.678813 15622 raft_consensus.cc:3037] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.681319 16241 consensus_queue.cc:1035] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.685802 16241 consensus_queue.cc:1035] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.694104 16511 raft_consensus.cc:482] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:13.694238 16511 raft_consensus.cc:504] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.694866 16511 leader_election.cc:288] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:13.695197 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1702f0e988804a9f930900ed4ab266fe" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:13.695256 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1702f0e988804a9f930900ed4ab266fe" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:13.695390 15622 raft_consensus.cc:2454] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:13.695449 12992 raft_consensus.cc:2454] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:13.695747 15822 leader_election.cc:302] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:13.696002 16511 raft_consensus.cc:2781] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:13.696110 16511 raft_consensus.cc:482] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:13.696219 16511 raft_consensus.cc:3037] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.697940 16511 raft_consensus.cc:504] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.698486 16511 leader_election.cc:288] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:13.698797 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1702f0e988804a9f930900ed4ab266fe" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:13.698971 15622 raft_consensus.cc:3037] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.698935 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1702f0e988804a9f930900ed4ab266fe" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:13.699090 12992 raft_consensus.cc:3037] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.700860 15622 raft_consensus.cc:2454] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:13.700994 12992 raft_consensus.cc:2454] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:13.701274 15822 leader_election.cc:302] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:13.701548 16511 raft_consensus.cc:2781] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:13.701905 16511 raft_consensus.cc:686] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:13.702206 16511 consensus_queue.cc:237] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.704756 12579 catalog_manager.cc:5484] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:13.746708 16241 consensus_queue.cc:1035] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.751982 16241 consensus_queue.cc:1035] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.764889 16451 consensus_queue.cc:1035] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.769229 16580 consensus_queue.cc:1035] T 432a6e076bf4401bb81903d91de38a57 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.820778 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:13.826195 15622 raft_consensus.cc:1261] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.826287 12992 raft_consensus.cc:1261] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.826822 16511 consensus_queue.cc:1035] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.827145 16575 consensus_queue.cc:1035] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.836964 12579 catalog_manager.cc:5576] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983) does not have the latest schema for tablet 1702f0e988804a9f930900ed4ab266fe (table rebalancer_dynamic_table [id=144b219c4b0b46c9a8e3d50f0d424847]). Expected version 1 got 0
I20240502 08:18:13.841171 12583 catalog_manager.cc:5576] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757) does not have the latest schema for tablet 1702f0e988804a9f930900ed4ab266fe (table rebalancer_dynamic_table [id=144b219c4b0b46c9a8e3d50f0d424847]). Expected version 1 got 0
I20240502 08:18:13.842670 16588 tablet.cc:1722] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:13.843600 16588 tablet.cc:1620] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:13.843640 16590 tablet.cc:1722] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:13.844264 16587 tablet.cc:1722] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:13.844867 16587 tablet.cc:1620] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:13.845234 12583 catalog_manager.cc:5576] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983) does not have the latest schema for tablet 1702f0e988804a9f930900ed4ab266fe (table rebalancer_dynamic_table [id=144b219c4b0b46c9a8e3d50f0d424847]). Expected version 1 got 0
I20240502 08:18:13.848825 16590 tablet.cc:1620] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:13.850982 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=144b219c4b0b46c9a8e3d50f0d424847] alter complete (version 1)
I20240502 08:18:13.920636 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:13.928757 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3592822ca849b9a47c4320b84754ae (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=320abaf9751a4925a5200feff7f4eacc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.929177 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3592822ca849b9a47c4320b84754ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.929661 15601 tablet_service.cc:1467] Processing CreateTablet for tablet e9081ea0b98443d38f42c14cd626f0d0 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=320abaf9751a4925a5200feff7f4eacc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.930079 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9081ea0b98443d38f42c14cd626f0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.931104 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 01b4100f5ab3405cabd93cff75f61822 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=320abaf9751a4925a5200feff7f4eacc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.931514 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b4100f5ab3405cabd93cff75f61822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.932186 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 01b4100f5ab3405cabd93cff75f61822 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=320abaf9751a4925a5200feff7f4eacc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.932518 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b4100f5ab3405cabd93cff75f61822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.934823 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3592822ca849b9a47c4320b84754ae (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=320abaf9751a4925a5200feff7f4eacc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.935187 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3592822ca849b9a47c4320b84754ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.936168 16591 tablet_bootstrap.cc:492] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:13.936396 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3592822ca849b9a47c4320b84754ae (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=320abaf9751a4925a5200feff7f4eacc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.936830 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3592822ca849b9a47c4320b84754ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.937852 16591 tablet_bootstrap.cc:654] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.938882 12842 tablet_service.cc:1467] Processing CreateTablet for tablet e9081ea0b98443d38f42c14cd626f0d0 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=320abaf9751a4925a5200feff7f4eacc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.939286 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9081ea0b98443d38f42c14cd626f0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.943506 15468 tablet_service.cc:1467] Processing CreateTablet for tablet e9081ea0b98443d38f42c14cd626f0d0 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=320abaf9751a4925a5200feff7f4eacc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.943874 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9081ea0b98443d38f42c14cd626f0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.945421 15466 tablet_service.cc:1467] Processing CreateTablet for tablet 01b4100f5ab3405cabd93cff75f61822 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=320abaf9751a4925a5200feff7f4eacc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.945782 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b4100f5ab3405cabd93cff75f61822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.955552 16595 tablet_bootstrap.cc:492] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:13.957041 16595 tablet_bootstrap.cc:654] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.958914 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:13.961808 16596 tablet_bootstrap.cc:492] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:13.963719 16594 tablet_bootstrap.cc:492] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:13.964210 16593 tablet_bootstrap.cc:492] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:13.966058 16594 tablet_bootstrap.cc:654] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.966686 16596 tablet_bootstrap.cc:654] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.967335 16593 tablet_bootstrap.cc:654] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.968103 16595 tablet_bootstrap.cc:492] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:13.968209 16591 tablet_bootstrap.cc:492] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:13.968279 16595 ts_tablet_manager.cc:1367] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 08:18:13.968432 16591 ts_tablet_manager.cc:1367] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.007s
I20240502 08:18:13.968952 16587 tablet.cc:1722] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:13.969115 16595 raft_consensus.cc:348] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.969307 16595 raft_consensus.cc:374] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.969424 16587 tablet.cc:1620] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:13.969440 16595 raft_consensus.cc:729] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.969370 16591 raft_consensus.cc:348] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:13.969677 16591 raft_consensus.cc:374] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.969709 16595 consensus_queue.cc:260] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.969947 16591 raft_consensus.cc:729] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.970291 16595 ts_tablet_manager.cc:1398] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:13.970222 16591 consensus_queue.cc:260] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:13.970888 16591 ts_tablet_manager.cc:1398] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:13.972010 16591 tablet_bootstrap.cc:492] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:13.973723 16591 tablet_bootstrap.cc:654] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.980543 16593 tablet_bootstrap.cc:492] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:13.980738 16593 ts_tablet_manager.cc:1367] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240502 08:18:13.981808 16593 raft_consensus.cc:348] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:13.982278 16593 raft_consensus.cc:374] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.982481 16593 raft_consensus.cc:729] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.982712 16591 tablet_bootstrap.cc:492] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:13.982787 16588 tablet.cc:1722] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:13.982887 16591 ts_tablet_manager.cc:1367] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240502 08:18:13.982841 16593 consensus_queue.cc:260] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:13.983273 16588 tablet.cc:1620] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:13.983572 16593 ts_tablet_manager.cc:1398] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:13.984486 16596 tablet_bootstrap.cc:492] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:13.984756 16596 ts_tablet_manager.cc:1367] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.003s
I20240502 08:18:13.986387 16596 raft_consensus.cc:348] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:13.986689 16596 raft_consensus.cc:374] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.986887 16596 raft_consensus.cc:729] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.987174 16596 consensus_queue.cc:260] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:13.988116 16596 ts_tablet_manager.cc:1398] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:13.988446 16593 tablet_bootstrap.cc:492] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:13.989905 16591 raft_consensus.cc:348] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.990120 16591 raft_consensus.cc:374] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.990211 16591 raft_consensus.cc:729] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.990494 16591 consensus_queue.cc:260] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:13.991097 16593 tablet_bootstrap.cc:654] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.991700 16591 ts_tablet_manager.cc:1398] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 08:18:13.992839 16593 tablet_bootstrap.cc:492] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:13.993240 16593 ts_tablet_manager.cc:1367] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 08:18:13.994035 16593 raft_consensus.cc:348] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:13.994339 16593 raft_consensus.cc:374] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.994486 16593 raft_consensus.cc:729] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.994772 16593 consensus_queue.cc:260] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:13.995326 16593 ts_tablet_manager.cc:1398] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:13.999989 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=144b219c4b0b46c9a8e3d50f0d424847] alter complete (version 2)
I20240502 08:18:14.008205 16594 tablet_bootstrap.cc:492] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:14.008365 16594 ts_tablet_manager.cc:1367] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.028s
I20240502 08:18:14.009110 16594 raft_consensus.cc:348] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.009312 16594 raft_consensus.cc:374] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.009402 16594 raft_consensus.cc:729] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.009624 16594 consensus_queue.cc:260] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.010200 16594 ts_tablet_manager.cc:1398] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:14.009866 16590 tablet.cc:1722] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:14.010553 16594 tablet_bootstrap.cc:492] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:14.010802 16590 tablet.cc:1620] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:14.012374 16594 tablet_bootstrap.cc:654] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.020867 16594 tablet_bootstrap.cc:492] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:14.021018 16594 ts_tablet_manager.cc:1367] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240502 08:18:14.021667 16594 raft_consensus.cc:348] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:14.021874 16594 raft_consensus.cc:374] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.021950 16594 raft_consensus.cc:729] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.022190 16594 consensus_queue.cc:260] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:14.022725 16594 ts_tablet_manager.cc:1398] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:14.023034 16594 tablet_bootstrap.cc:492] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:14.024428 16594 tablet_bootstrap.cc:654] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.035498 16594 tablet_bootstrap.cc:492] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:14.035670 16594 ts_tablet_manager.cc:1367] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.012s
I20240502 08:18:14.036412 16594 raft_consensus.cc:348] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:14.036612 16594 raft_consensus.cc:374] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.036793 16594 raft_consensus.cc:729] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.037009 16594 consensus_queue.cc:260] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:14.037483 16594 ts_tablet_manager.cc:1398] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:14.050124 16549 raft_consensus.cc:482] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.050299 16549 raft_consensus.cc:504] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:14.051290 16549 leader_election.cc:288] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:14.051628 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3592822ca849b9a47c4320b84754ae" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:14.051865 15487 raft_consensus.cc:2454] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:14.052351 12926 leader_election.cc:302] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:14.052675 16549 raft_consensus.cc:2781] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.052811 16549 raft_consensus.cc:482] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.052924 16549 raft_consensus.cc:3037] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.056133 16549 raft_consensus.cc:504] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:14.057602 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3592822ca849b9a47c4320b84754ae" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:14.057821 15487 raft_consensus.cc:3037] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.060346 15487 raft_consensus.cc:2454] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:14.060699 12926 leader_election.cc:302] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:14.061319 16607 raft_consensus.cc:2781] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.061429 16549 leader_election.cc:288] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:14.061980 16607 raft_consensus.cc:686] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:14.062314 16607 consensus_queue.cc:237] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:14.065469 12583 catalog_manager.cc:5484] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:14.067384 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3592822ca849b9a47c4320b84754ae" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:14.067490 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3592822ca849b9a47c4320b84754ae" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:14.067612 15886 raft_consensus.cc:2454] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:14.104647 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:14.112756 16587 tablet.cc:1722] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:14.113091 16590 tablet.cc:1722] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:14.113642 16588 tablet.cc:1722] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:14.114858 16582 raft_consensus.cc:482] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.115013 16582 raft_consensus.cc:504] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.115746 16582 leader_election.cc:288] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:14.116576 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9081ea0b98443d38f42c14cd626f0d0" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:14.116605 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9081ea0b98443d38f42c14cd626f0d0" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:14.116825 12861 raft_consensus.cc:2454] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:14.116839 15487 raft_consensus.cc:2454] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:14.117285 15554 leader_election.cc:302] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:14.117970 16582 raft_consensus.cc:2781] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.118084 16582 raft_consensus.cc:482] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.118183 16582 raft_consensus.cc:3037] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.121397 16582 raft_consensus.cc:504] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.121927 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=144b219c4b0b46c9a8e3d50f0d424847] alter complete (version 3)
I20240502 08:18:14.122438 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9081ea0b98443d38f42c14cd626f0d0" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:14.122646 15487 raft_consensus.cc:3037] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.124521 15487 raft_consensus.cc:2454] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:14.124856 15554 leader_election.cc:302] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:14.125098 16451 raft_consensus.cc:2781] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.125258 16451 raft_consensus.cc:686] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:14.125497 16451 consensus_queue.cc:237] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.126845 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9081ea0b98443d38f42c14cd626f0d0" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:14.127066 12861 raft_consensus.cc:3037] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.128458 12579 catalog_manager.cc:5484] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:14.129249 12861 raft_consensus.cc:2454] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:14.126395 16582 leader_election.cc:288] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:14.227761 12579 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:14.227996 12579 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:14.230846 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 1702f0e988804a9f930900ed4ab266fe
I20240502 08:18:14.231624 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 1702f0e988804a9f930900ed4ab266fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:14 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:14.231758 15601 tablet_service.cc:1514] Processing DeleteTablet for tablet 1702f0e988804a9f930900ed4ab266fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:14 UTC) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:14.232903 16612 tablet_replica.cc:330] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:14.232975 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 1702f0e988804a9f930900ed4ab266fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:14 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:14.233175 16612 raft_consensus.cc:2229] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:14.233650 16612 raft_consensus.cc:2258] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:14.234436 16612 ts_tablet_manager.cc:1875] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:14.240674 16612 ts_tablet_manager.cc:1888] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:14.240803 16612 log.cc:1196] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/1702f0e988804a9f930900ed4ab266fe
I20240502 08:18:14.241755 16612 ts_tablet_manager.cc:1909] T 1702f0e988804a9f930900ed4ab266fe P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:14.242787 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 1702f0e988804a9f930900ed4ab266fe (table rebalancer_dynamic_table_new_name [id=144b219c4b0b46c9a8e3d50f0d424847]) successfully deleted
I20240502 08:18:14.242983 16614 tablet_replica.cc:330] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:14.243160 16614 raft_consensus.cc:2229] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:14.243175 16613 tablet_replica.cc:330] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:14.243326 16613 raft_consensus.cc:2229] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:14.243373 16614 raft_consensus.cc:2258] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:14.243541 16613 raft_consensus.cc:2258] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:14.244086 16614 ts_tablet_manager.cc:1875] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:14.244350 16613 ts_tablet_manager.cc:1875] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:14.248551 16614 ts_tablet_manager.cc:1888] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:14.248664 16614 log.cc:1196] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/1702f0e988804a9f930900ed4ab266fe
I20240502 08:18:14.248991 16613 ts_tablet_manager.cc:1888] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:14.249069 16614 ts_tablet_manager.cc:1909] T 1702f0e988804a9f930900ed4ab266fe P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:14.249114 16613 log.cc:1196] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/1702f0e988804a9f930900ed4ab266fe
I20240502 08:18:14.249455 16613 ts_tablet_manager.cc:1909] T 1702f0e988804a9f930900ed4ab266fe P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting consensus metadata
I20240502 08:18:14.249926 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 1702f0e988804a9f930900ed4ab266fe (table rebalancer_dynamic_table_new_name [id=144b219c4b0b46c9a8e3d50f0d424847]) successfully deleted
I20240502 08:18:14.250303 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 1702f0e988804a9f930900ed4ab266fe (table rebalancer_dynamic_table_new_name [id=144b219c4b0b46c9a8e3d50f0d424847]) successfully deleted
I20240502 08:18:14.269274 16575 raft_consensus.cc:482] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.269465 16575 raft_consensus.cc:504] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:14.270239 16575 leader_election.cc:288] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:14.270944 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b4100f5ab3405cabd93cff75f61822" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:14.271165 15622 raft_consensus.cc:2454] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:14.272336 15822 leader_election.cc:302] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:14.272635 16575 raft_consensus.cc:2781] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.272728 16575 raft_consensus.cc:482] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.272822 16575 raft_consensus.cc:3037] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.274667 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b4100f5ab3405cabd93cff75f61822" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:14.274879 15487 raft_consensus.cc:2454] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:14.275503 16575 raft_consensus.cc:504] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:14.276144 16575 leader_election.cc:288] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:14.276485 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b4100f5ab3405cabd93cff75f61822" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:14.276680 15487 raft_consensus.cc:3037] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.276666 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b4100f5ab3405cabd93cff75f61822" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:14.276854 15622 raft_consensus.cc:3037] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.278940 15622 raft_consensus.cc:2454] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:14.279302 15822 leader_election.cc:302] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:14.279460 15487 raft_consensus.cc:2454] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:14.279546 16575 raft_consensus.cc:2781] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.279695 16575 raft_consensus.cc:686] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:14.279963 16575 consensus_queue.cc:237] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:14.283123 12579 catalog_manager.cc:5484] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:14.333621 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:14.341838 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 320e7c9770c7404c95e7c7e467d58682 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e3d202813d3f4d4eb12e4dc1f4b22e12]), partition=
I20240502 08:18:14.341854 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 320e7c9770c7404c95e7c7e467d58682 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e3d202813d3f4d4eb12e4dc1f4b22e12]), partition=
I20240502 08:18:14.342145 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 320e7c9770c7404c95e7c7e467d58682 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e3d202813d3f4d4eb12e4dc1f4b22e12]), partition=
I20240502 08:18:14.342212 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320e7c9770c7404c95e7c7e467d58682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.342224 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320e7c9770c7404c95e7c7e467d58682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.342494 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320e7c9770c7404c95e7c7e467d58682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.348258 16594 tablet_bootstrap.cc:492] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:14.348508 16591 tablet_bootstrap.cc:492] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:14.348680 16593 tablet_bootstrap.cc:492] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:14.349753 16594 tablet_bootstrap.cc:654] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.349882 16591 tablet_bootstrap.cc:654] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.350100 16593 tablet_bootstrap.cc:654] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.351941 16594 tablet_bootstrap.cc:492] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:14.352103 16594 ts_tablet_manager.cc:1367] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240502 08:18:14.352569 16593 tablet_bootstrap.cc:492] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:14.352718 16593 ts_tablet_manager.cc:1367] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:14.352824 16594 raft_consensus.cc:348] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.353039 16594 raft_consensus.cc:374] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.353053 16591 tablet_bootstrap.cc:492] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:14.353103 16594 raft_consensus.cc:729] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.353149 16591 ts_tablet_manager.cc:1367] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:18:14.353348 16594 consensus_queue.cc:260] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.353477 16593 raft_consensus.cc:348] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.353642 16593 raft_consensus.cc:374] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.353724 16593 raft_consensus.cc:729] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.353909 16594 ts_tablet_manager.cc:1398] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:14.353960 16593 consensus_queue.cc:260] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.353667 16591 raft_consensus.cc:348] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.354662 16593 ts_tablet_manager.cc:1398] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:14.354714 16591 raft_consensus.cc:374] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.354808 16591 raft_consensus.cc:729] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.355033 16591 consensus_queue.cc:260] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.356456 16591 ts_tablet_manager.cc:1398] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:14.420660 16582 raft_consensus.cc:482] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.420822 16582 raft_consensus.cc:504] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.421430 16582 leader_election.cc:288] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:14.421872 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320e7c9770c7404c95e7c7e467d58682" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:14.421969 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320e7c9770c7404c95e7c7e467d58682" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:14.422070 15886 raft_consensus.cc:2454] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:14.422227 15487 raft_consensus.cc:2454] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:14.422544 15555 leader_election.cc:302] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:14.422838 16582 raft_consensus.cc:2781] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.422967 16582 raft_consensus.cc:482] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.423058 16582 raft_consensus.cc:3037] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.426015 16582 raft_consensus.cc:504] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.427057 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320e7c9770c7404c95e7c7e467d58682" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:14.427115 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320e7c9770c7404c95e7c7e467d58682" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:14.427248 15886 raft_consensus.cc:3037] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.427313 15487 raft_consensus.cc:3037] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.429136 15886 raft_consensus.cc:2454] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:14.429530 15555 leader_election.cc:302] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:14.429816 16451 raft_consensus.cc:2781] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.429957 16451 raft_consensus.cc:686] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:14.430001 15487 raft_consensus.cc:2454] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:14.430208 16451 consensus_queue.cc:237] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.430863 16582 leader_election.cc:288] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:14.433259 12579 catalog_manager.cc:5484] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:14.483287 15886 raft_consensus.cc:3037] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.486022 16607 consensus_queue.cc:1035] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.490569 16608 consensus_queue.cc:1035] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.544776 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:14.545370 16451 consensus_queue.cc:1035] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.550890 16451 consensus_queue.cc:1035] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.551848 15886 raft_consensus.cc:1261] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:14.552423 16451 consensus_queue.cc:1035] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.554440 15487 raft_consensus.cc:1261] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:14.555676 16451 consensus_queue.cc:1035] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.556406 16588 tablet.cc:1722] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:14.556879 16588 tablet.cc:1620] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:14.557844 16587 tablet.cc:1722] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:14.558293 16587 tablet.cc:1620] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:14.558271 12579 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 320e7c9770c7404c95e7c7e467d58682 (table rebalancer_dynamic_table [id=e3d202813d3f4d4eb12e4dc1f4b22e12]). Expected version 1 got 0
I20240502 08:18:14.563844 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet 320e7c9770c7404c95e7c7e467d58682 (table rebalancer_dynamic_table [id=e3d202813d3f4d4eb12e4dc1f4b22e12]). Expected version 1 got 0
I20240502 08:18:14.565227 16629 tablet.cc:1722] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:14.566568 16629 tablet.cc:1620] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:14.567474 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet 320e7c9770c7404c95e7c7e467d58682 (table rebalancer_dynamic_table [id=e3d202813d3f4d4eb12e4dc1f4b22e12]). Expected version 1 got 0
I20240502 08:18:14.568778 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=e3d202813d3f4d4eb12e4dc1f4b22e12] alter complete (version 1)
I20240502 08:18:14.672274 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:14.678637 16588 tablet.cc:1722] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:14.679193 16588 tablet.cc:1620] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:14.679634 16587 tablet.cc:1722] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:14.680034 16587 tablet.cc:1620] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:14.680239 16629 tablet.cc:1722] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:14.681103 16629 tablet.cc:1620] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:14.688467 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=e3d202813d3f4d4eb12e4dc1f4b22e12] alter complete (version 2)
I20240502 08:18:14.719179 16511 consensus_queue.cc:1035] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.723234 16632 consensus_queue.cc:1035] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.791842 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:14.796289 12583 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:14.799310 16588 tablet.cc:1722] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:14.800457 16629 tablet.cc:1722] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:14.806942 16587 tablet.cc:1722] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:14.813354 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=e3d202813d3f4d4eb12e4dc1f4b22e12] alter complete (version 3)
I20240502 08:18:14.817770 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcbd19e0a9b49eaa3e403683fe6b34b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=6a2ad006f2a4419ebb5b776a03711658]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.817955 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcbd19e0a9b49eaa3e403683fe6b34b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=6a2ad006f2a4419ebb5b776a03711658]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.818189 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcbd19e0a9b49eaa3e403683fe6b34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.818364 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcbd19e0a9b49eaa3e403683fe6b34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.822983 16594 tablet_bootstrap.cc:492] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:14.823784 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcbd19e0a9b49eaa3e403683fe6b34b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=6a2ad006f2a4419ebb5b776a03711658]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.824206 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcbd19e0a9b49eaa3e403683fe6b34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.824573 16594 tablet_bootstrap.cc:654] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.826187 16591 tablet_bootstrap.cc:492] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:14.827889 16591 tablet_bootstrap.cc:654] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.828488 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3814d6b89048558957930967db0178 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=6a2ad006f2a4419ebb5b776a03711658]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.828924 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3814d6b89048558957930967db0178. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.829226 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3814d6b89048558957930967db0178 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=6a2ad006f2a4419ebb5b776a03711658]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.829675 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3814d6b89048558957930967db0178. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.830273 16593 tablet_bootstrap.cc:492] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:14.831753 16593 tablet_bootstrap.cc:654] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.835717 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3814d6b89048558957930967db0178 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=6a2ad006f2a4419ebb5b776a03711658]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.836127 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3814d6b89048558957930967db0178. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.837785 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 5f705e37c6be455eaa2a19145474bf9e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=6a2ad006f2a4419ebb5b776a03711658]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.838164 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f705e37c6be455eaa2a19145474bf9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.840564 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 5f705e37c6be455eaa2a19145474bf9e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=6a2ad006f2a4419ebb5b776a03711658]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.840934 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f705e37c6be455eaa2a19145474bf9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.841853 16594 tablet_bootstrap.cc:492] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:14.842025 16594 ts_tablet_manager.cc:1367] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240502 08:18:14.842989 16594 raft_consensus.cc:348] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.843231 16594 raft_consensus.cc:374] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.843335 16594 raft_consensus.cc:729] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.843565 16594 consensus_queue.cc:260] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.844216 16594 ts_tablet_manager.cc:1398] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:14.847784 16594 tablet_bootstrap.cc:492] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:14.849439 16594 tablet_bootstrap.cc:654] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.853600 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 5f705e37c6be455eaa2a19145474bf9e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=6a2ad006f2a4419ebb5b776a03711658]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.854012 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f705e37c6be455eaa2a19145474bf9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.858876 16638 tablet_bootstrap.cc:492] T 2e3814d6b89048558957930967db0178 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:14.860358 16640 tablet_bootstrap.cc:492] T 2e3814d6b89048558957930967db0178 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:14.865614 16591 tablet_bootstrap.cc:492] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:14.865945 16591 ts_tablet_manager.cc:1367] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.007s
I20240502 08:18:14.866940 16591 raft_consensus.cc:348] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.867286 16591 raft_consensus.cc:374] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.867440 16591 raft_consensus.cc:729] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.867774 16591 consensus_queue.cc:260] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.868546 16591 ts_tablet_manager.cc:1398] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:14.869984 16638 tablet_bootstrap.cc:654] T 2e3814d6b89048558957930967db0178 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.870684 16640 tablet_bootstrap.cc:654] T 2e3814d6b89048558957930967db0178 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.873005 16637 tablet_bootstrap.cc:492] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:14.874989 16637 tablet_bootstrap.cc:654] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.875221 16594 tablet_bootstrap.cc:492] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:14.875420 16594 ts_tablet_manager.cc:1367] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.006s
I20240502 08:18:14.876137 16640 tablet_bootstrap.cc:492] T 2e3814d6b89048558957930967db0178 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:14.876085 16594 raft_consensus.cc:348] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.876294 16640 ts_tablet_manager.cc:1367] T 2e3814d6b89048558957930967db0178 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20240502 08:18:14.876295 16594 raft_consensus.cc:374] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.876374 16594 raft_consensus.cc:729] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.876762 16593 tablet_bootstrap.cc:492] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:14.876621 16594 consensus_queue.cc:260] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.876933 16593 ts_tablet_manager.cc:1367] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.001s
I20240502 08:18:14.877046 16640 raft_consensus.cc:348] T 2e3814d6b89048558957930967db0178 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:14.877184 16594 ts_tablet_manager.cc:1398] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:14.877267 16640 raft_consensus.cc:374] T 2e3814d6b89048558957930967db0178 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.877372 16640 raft_consensus.cc:729] T 2e3814d6b89048558957930967db0178 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.877645 16640 consensus_queue.cc:260] T 2e3814d6b89048558957930967db0178 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:14.879281 16638 tablet_bootstrap.cc:492] T 2e3814d6b89048558957930967db0178 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:14.879786 16638 ts_tablet_manager.cc:1367] T 2e3814d6b89048558957930967db0178 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20240502 08:18:14.880477 16638 raft_consensus.cc:348] T 2e3814d6b89048558957930967db0178 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:14.879781 16593 raft_consensus.cc:348] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.880755 16593 raft_consensus.cc:374] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.880801 16638 raft_consensus.cc:374] T 2e3814d6b89048558957930967db0178 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.881003 16638 raft_consensus.cc:729] T 2e3814d6b89048558957930967db0178 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.881273 16638 consensus_queue.cc:260] T 2e3814d6b89048558957930967db0178 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:14.881880 16638 ts_tablet_manager.cc:1398] T 2e3814d6b89048558957930967db0178 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:14.880856 16593 raft_consensus.cc:729] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.883106 16640 ts_tablet_manager.cc:1398] T 2e3814d6b89048558957930967db0178 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:18:14.884320 16637 tablet_bootstrap.cc:492] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:14.884467 16637 ts_tablet_manager.cc:1367] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240502 08:18:14.885207 16637 raft_consensus.cc:348] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:14.885391 16637 raft_consensus.cc:374] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.885493 16637 raft_consensus.cc:729] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.882750 16593 consensus_queue.cc:260] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.885812 16637 consensus_queue.cc:260] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:14.886644 16637 ts_tablet_manager.cc:1398] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:14.887037 16637 tablet_bootstrap.cc:492] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:14.888746 16637 tablet_bootstrap.cc:654] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.890321 16637 tablet_bootstrap.cc:492] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:14.890498 16637 ts_tablet_manager.cc:1367] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 08:18:14.891024 16593 ts_tablet_manager.cc:1398] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240502 08:18:14.891178 16637 raft_consensus.cc:348] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.891332 16593 tablet_bootstrap.cc:492] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:14.891372 16637 raft_consensus.cc:374] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.891443 16637 raft_consensus.cc:729] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.891695 16637 consensus_queue.cc:260] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.892175 16637 ts_tablet_manager.cc:1398] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:14.894536 16593 tablet_bootstrap.cc:654] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.896299 16593 tablet_bootstrap.cc:492] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:14.896425 16593 ts_tablet_manager.cc:1367] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:18:14.896975 16593 raft_consensus.cc:348] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.897118 16593 raft_consensus.cc:374] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.897377 16593 raft_consensus.cc:729] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.897588 16593 consensus_queue.cc:260] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.898074 16593 ts_tablet_manager.cc:1398] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:14.900297 16580 raft_consensus.cc:482] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.900486 16580 raft_consensus.cc:504] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.901086 16580 leader_election.cc:288] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:14.901472 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcbd19e0a9b49eaa3e403683fe6b34b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:14.901717 15886 raft_consensus.cc:2454] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:14.902071 15555 leader_election.cc:302] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:14.902346 16580 raft_consensus.cc:2781] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.902452 16580 raft_consensus.cc:482] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.902391 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcbd19e0a9b49eaa3e403683fe6b34b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:14.902575 16580 raft_consensus.cc:3037] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.902629 15487 raft_consensus.cc:2454] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:14.904497 16580 raft_consensus.cc:504] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.905051 16580 leader_election.cc:288] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:14.905406 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcbd19e0a9b49eaa3e403683fe6b34b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:14.905647 15886 raft_consensus.cc:3037] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.905731 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcbd19e0a9b49eaa3e403683fe6b34b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:14.905872 15487 raft_consensus.cc:3037] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.907797 15886 raft_consensus.cc:2454] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:14.907938 15487 raft_consensus.cc:2454] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:14.908176 15555 leader_election.cc:302] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:14.908430 16580 raft_consensus.cc:2781] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.908594 16580 raft_consensus.cc:686] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:14.908851 16580 consensus_queue.cc:237] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:14.911610 12587 catalog_manager.cc:5484] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:14.917037 12587 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:14.917207 12587 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:14.919816 12587 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 320e7c9770c7404c95e7c7e467d58682
I20240502 08:18:14.920712 15601 tablet_service.cc:1514] Processing DeleteTablet for tablet 320e7c9770c7404c95e7c7e467d58682 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:14 UTC) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:14.921001 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 320e7c9770c7404c95e7c7e467d58682 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:14 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:14.921316 16648 tablet_replica.cc:330] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:14.921499 16648 raft_consensus.cc:2229] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:14.921533 16649 tablet_replica.cc:330] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:14.921706 16649 raft_consensus.cc:2229] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:14.921829 16648 raft_consensus.cc:2258] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:14.922017 16649 raft_consensus.cc:2258] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:14.922173 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet 320e7c9770c7404c95e7c7e467d58682 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:14 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:14.922559 16648 ts_tablet_manager.cc:1875] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:14.922719 16650 tablet_replica.cc:330] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:14.922904 16650 raft_consensus.cc:2229] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:14.922955 16649 ts_tablet_manager.cc:1875] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:14.923087 16650 raft_consensus.cc:2258] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:14.923808 16650 ts_tablet_manager.cc:1875] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:14.928257 16650 ts_tablet_manager.cc:1888] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:14.928375 16650 log.cc:1196] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/320e7c9770c7404c95e7c7e467d58682
I20240502 08:18:14.928730 16650 ts_tablet_manager.cc:1909] T 320e7c9770c7404c95e7c7e467d58682 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:14.928792 16649 ts_tablet_manager.cc:1888] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:14.928910 16649 log.cc:1196] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/320e7c9770c7404c95e7c7e467d58682
I20240502 08:18:14.929108 16648 ts_tablet_manager.cc:1888] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:14.929219 16648 log.cc:1196] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/320e7c9770c7404c95e7c7e467d58682
I20240502 08:18:14.929452 16649 ts_tablet_manager.cc:1909] T 320e7c9770c7404c95e7c7e467d58682 P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:14.929634 16648 ts_tablet_manager.cc:1909] T 320e7c9770c7404c95e7c7e467d58682 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting consensus metadata
I20240502 08:18:14.929611 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 320e7c9770c7404c95e7c7e467d58682 (table rebalancer_dynamic_table_new_name [id=e3d202813d3f4d4eb12e4dc1f4b22e12]) successfully deleted
I20240502 08:18:14.930385 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 320e7c9770c7404c95e7c7e467d58682 (table rebalancer_dynamic_table_new_name [id=e3d202813d3f4d4eb12e4dc1f4b22e12]) successfully deleted
I20240502 08:18:14.930473 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 320e7c9770c7404c95e7c7e467d58682 (table rebalancer_dynamic_table_new_name [id=e3d202813d3f4d4eb12e4dc1f4b22e12]) successfully deleted
I20240502 08:18:14.935180 16477 raft_consensus.cc:482] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.935313 16477 raft_consensus.cc:504] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:14.936101 16477 leader_election.cc:288] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:14.943127 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3814d6b89048558957930967db0178" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:14.943368 13119 raft_consensus.cc:2454] T 2e3814d6b89048558957930967db0178 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:14.943776 15687 leader_election.cc:302] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:14.944020 16477 raft_consensus.cc:2781] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.943951 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3814d6b89048558957930967db0178" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:14.944134 16477 raft_consensus.cc:482] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.944190 12730 raft_consensus.cc:2454] T 2e3814d6b89048558957930967db0178 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:14.944228 16477 raft_consensus.cc:3037] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.946028 16477 raft_consensus.cc:504] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:14.946538 16477 leader_election.cc:288] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:14.946866 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3814d6b89048558957930967db0178" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:14.947033 12730 raft_consensus.cc:3037] T 2e3814d6b89048558957930967db0178 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.948750 12730 raft_consensus.cc:2454] T 2e3814d6b89048558957930967db0178 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:14.949070 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3814d6b89048558957930967db0178" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:14.949101 15688 leader_election.cc:302] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:14.949254 13119 raft_consensus.cc:3037] T 2e3814d6b89048558957930967db0178 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.949385 16477 raft_consensus.cc:2781] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.949779 16477 raft_consensus.cc:686] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:14.950069 16477 consensus_queue.cc:237] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:14.951670 13119 raft_consensus.cc:2454] T 2e3814d6b89048558957930967db0178 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:14.952977 12587 catalog_manager.cc:5484] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:14.959398 16619 raft_consensus.cc:482] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.959532 16619 raft_consensus.cc:504] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.960307 16619 leader_election.cc:288] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:14.960691 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f705e37c6be455eaa2a19145474bf9e" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:14.960748 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f705e37c6be455eaa2a19145474bf9e" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:14.960912 15754 raft_consensus.cc:2454] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:14.960989 15622 raft_consensus.cc:2454] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:14.961318 15421 leader_election.cc:302] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:14.961608 16619 raft_consensus.cc:2781] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.961730 16619 raft_consensus.cc:482] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.961850 16619 raft_consensus.cc:3037] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.963959 16619 raft_consensus.cc:504] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.964479 16619 leader_election.cc:288] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:14.964778 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f705e37c6be455eaa2a19145474bf9e" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:14.964952 15754 raft_consensus.cc:3037] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.966729 15754 raft_consensus.cc:2454] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:14.967085 15418 leader_election.cc:302] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:14.967381 16619 raft_consensus.cc:2781] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.967506 16619 raft_consensus.cc:686] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:14.967588 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f705e37c6be455eaa2a19145474bf9e" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:14.967808 15622 raft_consensus.cc:3037] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.967765 16619 consensus_queue.cc:237] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:14.970208 15622 raft_consensus.cc:2454] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:14.970743 12587 catalog_manager.cc:5484] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:15.022946 12587 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:15.030436 15468 tablet_service.cc:1467] Processing CreateTablet for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8081800cb35147b4977f055fffac7eec]), partition=
I20240502 08:18:15.030624 13102 tablet_service.cc:1467] Processing CreateTablet for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8081800cb35147b4977f055fffac7eec]), partition=
I20240502 08:18:15.030849 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.031005 12970 tablet_service.cc:1467] Processing CreateTablet for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8081800cb35147b4977f055fffac7eec]), partition=
I20240502 08:18:15.031031 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.031301 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.035957 16638 tablet_bootstrap.cc:492] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:15.036520 16594 tablet_bootstrap.cc:492] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:15.037528 16638 tablet_bootstrap.cc:654] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.037905 16594 tablet_bootstrap.cc:654] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.048208 16594 tablet_bootstrap.cc:492] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:15.048377 16594 ts_tablet_manager.cc:1367] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.001s
I20240502 08:18:15.049175 16638 tablet_bootstrap.cc:492] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:15.049207 16594 raft_consensus.cc:348] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.049410 16638 ts_tablet_manager.cc:1367] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20240502 08:18:15.049414 16594 raft_consensus.cc:374] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.049507 16594 raft_consensus.cc:729] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.049721 16594 consensus_queue.cc:260] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.050320 16594 ts_tablet_manager.cc:1398] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:15.051025 16661 tablet_bootstrap.cc:492] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:15.050940 16638 raft_consensus.cc:348] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.051241 16638 raft_consensus.cc:374] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.051343 16638 raft_consensus.cc:729] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.051633 16638 consensus_queue.cc:260] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.052687 16661 tablet_bootstrap.cc:654] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.054654 16638 ts_tablet_manager.cc:1398] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:15.056511 16661 tablet_bootstrap.cc:492] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:15.056735 16661 ts_tablet_manager.cc:1367] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240502 08:18:15.057909 16661 raft_consensus.cc:348] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.058341 16661 raft_consensus.cc:374] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.058513 16661 raft_consensus.cc:729] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.059442 16661 consensus_queue.cc:260] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.060031 16661 ts_tablet_manager.cc:1398] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:15.249902 16608 raft_consensus.cc:482] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.250104 16608 raft_consensus.cc:504] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.250907 16608 leader_election.cc:288] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:15.251243 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62c0ecaf8ac4fc4b03faee90c02a8c7" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:15.251477 15487 raft_consensus.cc:2454] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:15.251505 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62c0ecaf8ac4fc4b03faee90c02a8c7" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:15.251725 13119 raft_consensus.cc:2454] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:15.251870 12926 leader_election.cc:302] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:15.252171 16607 raft_consensus.cc:2781] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:15.252278 16607 raft_consensus.cc:482] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:15.252395 16607 raft_consensus.cc:3037] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.254221 16607 raft_consensus.cc:504] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.254833 16607 leader_election.cc:288] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:15.255164 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62c0ecaf8ac4fc4b03faee90c02a8c7" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:15.255379 15487 raft_consensus.cc:3037] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.255364 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62c0ecaf8ac4fc4b03faee90c02a8c7" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:15.255582 13121 raft_consensus.cc:3037] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.257630 15487 raft_consensus.cc:2454] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:15.257755 13121 raft_consensus.cc:2454] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:15.257998 12926 leader_election.cc:302] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:15.258255 16607 raft_consensus.cc:2781] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:15.258419 16607 raft_consensus.cc:686] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:15.258703 16607 consensus_queue.cc:237] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.261785 12587 catalog_manager.cc:5484] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:15.327088 16477 consensus_queue.cc:1035] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.329391 16580 consensus_queue.cc:1035] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.335808 16655 consensus_queue.cc:1035] T 2e3814d6b89048558957930967db0178 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.347448 16580 consensus_queue.cc:1035] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.374652 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:15.379768 13119 raft_consensus.cc:1261] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:15.379951 15487 raft_consensus.cc:1261] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:15.380307 16608 consensus_queue.cc:1035] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.380513 16607 consensus_queue.cc:1035] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.385810 12587 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 (table rebalancer_dynamic_table [id=8081800cb35147b4977f055fffac7eec]). Expected version 1 got 0
I20240502 08:18:15.385986 16679 tablet.cc:1722] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:15.386591 16679 tablet.cc:1620] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:15.388785 12587 catalog_manager.cc:5576] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979) does not have the latest schema for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 (table rebalancer_dynamic_table [id=8081800cb35147b4977f055fffac7eec]). Expected version 1 got 0
I20240502 08:18:15.390633 16680 tablet.cc:1722] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:15.391287 16680 tablet.cc:1620] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:15.392446 12579 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 (table rebalancer_dynamic_table [id=8081800cb35147b4977f055fffac7eec]). Expected version 1 got 0
I20240502 08:18:15.395699 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=8081800cb35147b4977f055fffac7eec] alter complete (version 1)
I20240502 08:18:15.399012 16681 tablet.cc:1722] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:15.400395 16681 tablet.cc:1620] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:15.401716 12579 catalog_manager.cc:5576] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979) does not have the latest schema for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 (table rebalancer_dynamic_table [id=8081800cb35147b4977f055fffac7eec]). Expected version 1 got 0
I20240502 08:18:15.414156 16657 consensus_queue.cc:1035] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.419473 16619 consensus_queue.cc:1035] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.488137 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:15.496412 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 5859ab9a10054123bcf6b74bdb2f93cc (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=59dbf2adb22b41a4b7b92d1b39c10d56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.496590 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 5859ab9a10054123bcf6b74bdb2f93cc (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=59dbf2adb22b41a4b7b92d1b39c10d56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.496867 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5859ab9a10054123bcf6b74bdb2f93cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.496788 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 5859ab9a10054123bcf6b74bdb2f93cc (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=59dbf2adb22b41a4b7b92d1b39c10d56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.497023 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5859ab9a10054123bcf6b74bdb2f93cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.497202 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5859ab9a10054123bcf6b74bdb2f93cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.498010 15734 tablet_service.cc:1467] Processing CreateTablet for tablet d6fdafe07a2846ebbea7a18f46c90794 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=59dbf2adb22b41a4b7b92d1b39c10d56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.498396 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fdafe07a2846ebbea7a18f46c90794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.499609 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 9a55a182e1f54db698ba8060eea80da0 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=59dbf2adb22b41a4b7b92d1b39c10d56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.499990 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a55a182e1f54db698ba8060eea80da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.502022 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 9a55a182e1f54db698ba8060eea80da0 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=59dbf2adb22b41a4b7b92d1b39c10d56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.502346 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a55a182e1f54db698ba8060eea80da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.503134 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 9a55a182e1f54db698ba8060eea80da0 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=59dbf2adb22b41a4b7b92d1b39c10d56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.503540 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a55a182e1f54db698ba8060eea80da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.506832 13102 tablet_service.cc:1467] Processing CreateTablet for tablet d6fdafe07a2846ebbea7a18f46c90794 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=59dbf2adb22b41a4b7b92d1b39c10d56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.507221 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fdafe07a2846ebbea7a18f46c90794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.512022 12972 tablet_service.cc:1467] Processing CreateTablet for tablet d6fdafe07a2846ebbea7a18f46c90794 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=59dbf2adb22b41a4b7b92d1b39c10d56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.512454 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6fdafe07a2846ebbea7a18f46c90794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.517508 16594 tablet_bootstrap.cc:492] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:15.517668 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:15.518446 16638 tablet_bootstrap.cc:492] T d6fdafe07a2846ebbea7a18f46c90794 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:15.519381 16594 tablet_bootstrap.cc:654] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.519973 16638 tablet_bootstrap.cc:654] T d6fdafe07a2846ebbea7a18f46c90794 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.522512 16661 tablet_bootstrap.cc:492] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:15.523950 16661 tablet_bootstrap.cc:654] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.525652 16661 tablet_bootstrap.cc:492] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:15.525817 16661 ts_tablet_manager.cc:1367] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:15.526234 16638 tablet_bootstrap.cc:492] T d6fdafe07a2846ebbea7a18f46c90794 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:15.526408 16638 ts_tablet_manager.cc:1367] T d6fdafe07a2846ebbea7a18f46c90794 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20240502 08:18:15.527220 16661 raft_consensus.cc:348] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.527200 16638 raft_consensus.cc:348] T d6fdafe07a2846ebbea7a18f46c90794 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.527436 16661 raft_consensus.cc:374] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.527511 16661 raft_consensus.cc:729] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.527545 16638 raft_consensus.cc:374] T d6fdafe07a2846ebbea7a18f46c90794 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.527741 16638 raft_consensus.cc:729] T d6fdafe07a2846ebbea7a18f46c90794 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.527827 16661 consensus_queue.cc:260] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.528043 16638 consensus_queue.cc:260] T d6fdafe07a2846ebbea7a18f46c90794 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.528352 16661 ts_tablet_manager.cc:1398] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:15.528661 16661 tablet_bootstrap.cc:492] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:15.528648 16638 ts_tablet_manager.cc:1398] T d6fdafe07a2846ebbea7a18f46c90794 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:15.530043 16661 tablet_bootstrap.cc:654] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.530419 16685 tablet_bootstrap.cc:492] T d6fdafe07a2846ebbea7a18f46c90794 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:15.532084 16685 tablet_bootstrap.cc:654] T d6fdafe07a2846ebbea7a18f46c90794 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.537825 16661 tablet_bootstrap.cc:492] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:15.537952 16661 ts_tablet_manager.cc:1367] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240502 08:18:15.538513 16594 tablet_bootstrap.cc:492] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:15.538702 16608 raft_consensus.cc:482] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.538645 16661 raft_consensus.cc:348] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.538831 16661 raft_consensus.cc:374] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.538933 16661 raft_consensus.cc:729] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.538866 16608 raft_consensus.cc:504] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.539147 16661 consensus_queue.cc:260] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.539561 16608 leader_election.cc:288] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:15.539673 16661 ts_tablet_manager.cc:1398] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:15.540346 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a55a182e1f54db698ba8060eea80da0" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:15.540560 16679 tablet.cc:1722] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:15.540961 16679 tablet.cc:1620] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
W20240502 08:18:15.541599 12926 leader_election.cc:341] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:15.542296 16680 tablet.cc:1722] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:15.542721 16680 tablet.cc:1620] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:15.545179 16594 ts_tablet_manager.cc:1367] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.005s
I20240502 08:18:15.546814 16594 raft_consensus.cc:348] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.547025 16594 raft_consensus.cc:374] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.547099 16594 raft_consensus.cc:729] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.548084 16594 consensus_queue.cc:260] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.549712 16685 tablet_bootstrap.cc:492] T d6fdafe07a2846ebbea7a18f46c90794 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:15.549880 16594 ts_tablet_manager.cc:1398] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:15.550159 16594 tablet_bootstrap.cc:492] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:15.550189 16685 ts_tablet_manager.cc:1367] T d6fdafe07a2846ebbea7a18f46c90794 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20240502 08:18:15.551331 16685 raft_consensus.cc:348] T d6fdafe07a2846ebbea7a18f46c90794 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.551478 16685 raft_consensus.cc:374] T d6fdafe07a2846ebbea7a18f46c90794 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.551563 16685 raft_consensus.cc:729] T d6fdafe07a2846ebbea7a18f46c90794 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.551774 16685 consensus_queue.cc:260] T d6fdafe07a2846ebbea7a18f46c90794 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.552296 16685 ts_tablet_manager.cc:1398] T d6fdafe07a2846ebbea7a18f46c90794 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:15.552682 16685 tablet_bootstrap.cc:492] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:15.553893 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=8081800cb35147b4977f055fffac7eec] alter complete (version 2)
I20240502 08:18:15.554061 16685 tablet_bootstrap.cc:654] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.557833 16594 tablet_bootstrap.cc:654] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.562758 16594 tablet_bootstrap.cc:492] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:15.562891 16594 ts_tablet_manager.cc:1367] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 08:18:15.563541 16594 raft_consensus.cc:348] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.563733 16594 raft_consensus.cc:374] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.563817 16594 raft_consensus.cc:729] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.564015 16594 consensus_queue.cc:260] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.567267 16681 tablet.cc:1722] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:15.567731 16681 tablet.cc:1620] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:15.570237 16594 ts_tablet_manager.cc:1398] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:15.577481 16684 tablet_bootstrap.cc:492] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:15.580694 16684 tablet_bootstrap.cc:654] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.580755 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a55a182e1f54db698ba8060eea80da0" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
W20240502 08:18:15.581261 12925 leader_election.cc:341] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:15.581388 12925 leader_election.cc:302] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4; no voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:15.581678 16608 raft_consensus.cc:2726] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:15.582363 16685 tablet_bootstrap.cc:492] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:15.582540 16685 ts_tablet_manager.cc:1367] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.020s
I20240502 08:18:15.583225 16685 raft_consensus.cc:348] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.583411 16685 raft_consensus.cc:374] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.583501 16685 raft_consensus.cc:729] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.583750 16685 consensus_queue.cc:260] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.585021 16685 ts_tablet_manager.cc:1398] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:15.585321 16685 tablet_bootstrap.cc:492] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:15.586087 16684 tablet_bootstrap.cc:492] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:15.586232 16684 ts_tablet_manager.cc:1367] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 08:18:15.586817 16684 raft_consensus.cc:348] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.587010 16684 raft_consensus.cc:374] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.587106 16684 raft_consensus.cc:729] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.587311 16685 tablet_bootstrap.cc:654] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.587375 16684 consensus_queue.cc:260] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.588057 16684 ts_tablet_manager.cc:1398] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:15.589334 16685 tablet_bootstrap.cc:492] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:15.589478 16685 ts_tablet_manager.cc:1367] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 08:18:15.590190 16685 raft_consensus.cc:348] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.590375 16685 raft_consensus.cc:374] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.590497 16685 raft_consensus.cc:729] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.590729 16685 consensus_queue.cc:260] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.591796 16685 ts_tablet_manager.cc:1398] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:15.662364 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:15.670432 16679 tablet.cc:1722] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:15.671190 16681 tablet.cc:1722] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:15.673450 16680 tablet.cc:1722] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:15.679781 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=8081800cb35147b4977f055fffac7eec] alter complete (version 3)
I20240502 08:18:15.719295 16619 raft_consensus.cc:482] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.719432 16619 raft_consensus.cc:504] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.719967 16619 leader_election.cc:288] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:15.720356 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5859ab9a10054123bcf6b74bdb2f93cc" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:15.720415 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5859ab9a10054123bcf6b74bdb2f93cc" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:15.720566 12859 raft_consensus.cc:2454] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:15.720635 15754 raft_consensus.cc:2454] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:15.720978 15419 leader_election.cc:302] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db7893bb29a047b3b82cc60bfd8930a3, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:15.721283 16619 raft_consensus.cc:2781] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:15.721400 16619 raft_consensus.cc:482] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:15.721468 16619 raft_consensus.cc:3037] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.723479 16619 raft_consensus.cc:504] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.724062 16619 leader_election.cc:288] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:15.724408 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5859ab9a10054123bcf6b74bdb2f93cc" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:15.724514 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5859ab9a10054123bcf6b74bdb2f93cc" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:15.724608 12859 raft_consensus.cc:3037] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.724704 15754 raft_consensus.cc:3037] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.726589 12859 raft_consensus.cc:2454] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:15.727269 15754 raft_consensus.cc:2454] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:15.727677 15418 leader_election.cc:302] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:15.727931 16619 raft_consensus.cc:2781] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:15.728070 16619 raft_consensus.cc:686] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:15.728353 16619 consensus_queue.cc:237] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.731006 12584 catalog_manager.cc:5484] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:15.781747 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:15.782038 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:15.784655 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet f62c0ecaf8ac4fc4b03faee90c02a8c7
I20240502 08:18:15.785394 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:15 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:15.785562 12972 tablet_service.cc:1514] Processing DeleteTablet for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:15 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:15.785548 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:15 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:15.790235 16698 tablet_replica.cc:330] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:15.790433 16698 raft_consensus.cc:2229] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:15.790743 16698 raft_consensus.cc:2258] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:15.790917 16700 tablet_replica.cc:330] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:15.791097 16699 tablet_replica.cc:330] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:15.791126 16700 raft_consensus.cc:2229] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:15.791337 16699 raft_consensus.cc:2229] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:15.791538 16699 raft_consensus.cc:2258] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:15.791613 16698 ts_tablet_manager.cc:1875] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:15.791647 16700 raft_consensus.cc:2258] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:15.792284 16699 ts_tablet_manager.cc:1875] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:15.792603 16700 ts_tablet_manager.cc:1875] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:15.796710 16699 ts_tablet_manager.cc:1888] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:15.796824 16699 log.cc:1196] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/f62c0ecaf8ac4fc4b03faee90c02a8c7
I20240502 08:18:15.797165 16699 ts_tablet_manager.cc:1909] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:15.797766 16698 ts_tablet_manager.cc:1888] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:15.797919 16698 log.cc:1196] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/f62c0ecaf8ac4fc4b03faee90c02a8c7
I20240502 08:18:15.798035 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 (table rebalancer_dynamic_table_new_name [id=8081800cb35147b4977f055fffac7eec]) successfully deleted
I20240502 08:18:15.798362 16698 ts_tablet_manager.cc:1909] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:15.798547 16700 ts_tablet_manager.cc:1888] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:15.798728 16700 log.cc:1196] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/f62c0ecaf8ac4fc4b03faee90c02a8c7
I20240502 08:18:15.799158 16700 ts_tablet_manager.cc:1909] T f62c0ecaf8ac4fc4b03faee90c02a8c7 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:15.799387 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 (table rebalancer_dynamic_table_new_name [id=8081800cb35147b4977f055fffac7eec]) successfully deleted
I20240502 08:18:15.800196 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet f62c0ecaf8ac4fc4b03faee90c02a8c7 (table rebalancer_dynamic_table_new_name [id=8081800cb35147b4977f055fffac7eec]) successfully deleted
I20240502 08:18:15.809839 16607 raft_consensus.cc:482] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.810003 16607 raft_consensus.cc:504] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.810644 16607 leader_election.cc:288] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:15.810984 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fdafe07a2846ebbea7a18f46c90794" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:15.811103 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fdafe07a2846ebbea7a18f46c90794" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:15.811201 15754 raft_consensus.cc:2454] T d6fdafe07a2846ebbea7a18f46c90794 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:15.811311 13119 raft_consensus.cc:2454] T d6fdafe07a2846ebbea7a18f46c90794 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:15.811691 12925 leader_election.cc:302] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:15.811946 16608 raft_consensus.cc:2781] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:15.812033 16608 raft_consensus.cc:482] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:15.812095 16608 raft_consensus.cc:3037] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.813751 16608 raft_consensus.cc:504] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.814208 16608 leader_election.cc:288] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:15.814591 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fdafe07a2846ebbea7a18f46c90794" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:15.814781 15754 raft_consensus.cc:3037] T d6fdafe07a2846ebbea7a18f46c90794 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.814769 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6fdafe07a2846ebbea7a18f46c90794" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:15.814970 13119 raft_consensus.cc:3037] T d6fdafe07a2846ebbea7a18f46c90794 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.816738 15754 raft_consensus.cc:2454] T d6fdafe07a2846ebbea7a18f46c90794 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:15.817088 12925 leader_election.cc:302] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:15.817364 16608 raft_consensus.cc:2781] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:15.817490 16608 raft_consensus.cc:686] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:15.817523 13119 raft_consensus.cc:2454] T d6fdafe07a2846ebbea7a18f46c90794 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:15.817757 16608 consensus_queue.cc:237] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:15.821046 12584 catalog_manager.cc:5484] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:15.887166 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:15.893774 12842 tablet_service.cc:1467] Processing CreateTablet for tablet a4ee60132711414b94627a2ebe4f2c0b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c2162eb2b6e041a6a20084fbfed02b99]), partition=
I20240502 08:18:15.894143 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ee60132711414b94627a2ebe4f2c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.894289 15468 tablet_service.cc:1467] Processing CreateTablet for tablet a4ee60132711414b94627a2ebe4f2c0b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c2162eb2b6e041a6a20084fbfed02b99]), partition=
I20240502 08:18:15.894589 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ee60132711414b94627a2ebe4f2c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.895170 15601 tablet_service.cc:1467] Processing CreateTablet for tablet a4ee60132711414b94627a2ebe4f2c0b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c2162eb2b6e041a6a20084fbfed02b99]), partition=
I20240502 08:18:15.895475 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ee60132711414b94627a2ebe4f2c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.898722 16594 tablet_bootstrap.cc:492] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:15.899856 16684 tablet_bootstrap.cc:492] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:15.899964 16594 tablet_bootstrap.cc:654] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.901343 16684 tablet_bootstrap.cc:654] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.901834 16594 tablet_bootstrap.cc:492] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:15.901862 16703 tablet_bootstrap.cc:492] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:15.901943 16594 ts_tablet_manager.cc:1367] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:18:15.902498 16619 raft_consensus.cc:482] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.902662 16619 raft_consensus.cc:504] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.902848 16594 raft_consensus.cc:348] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:15.903028 16594 raft_consensus.cc:374] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.903110 16594 raft_consensus.cc:729] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.903498 16703 tablet_bootstrap.cc:654] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.903393 16594 consensus_queue.cc:260] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:15.903687 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a55a182e1f54db698ba8060eea80da0" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:15.903820 16684 tablet_bootstrap.cc:492] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:15.903911 12992 raft_consensus.cc:2454] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:15.903966 16684 ts_tablet_manager.cc:1367] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 08:18:15.904023 16594 ts_tablet_manager.cc:1398] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:15.904707 16684 raft_consensus.cc:348] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:15.905016 16684 raft_consensus.cc:374] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.905128 15419 leader_election.cc:302] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:15.905160 16684 raft_consensus.cc:729] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.905400 16657 raft_consensus.cc:2781] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:15.905505 16657 raft_consensus.cc:482] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:15.905582 16657 raft_consensus.cc:3037] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.905843 16703 tablet_bootstrap.cc:492] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:15.905970 16703 ts_tablet_manager.cc:1367] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:15.906303 16684 consensus_queue.cc:260] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:15.906940 16703 raft_consensus.cc:348] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:15.907121 16703 raft_consensus.cc:374] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.907199 16703 raft_consensus.cc:729] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.907413 16703 consensus_queue.cc:260] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:15.907830 16684 ts_tablet_manager.cc:1398] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:15.907824 16657 raft_consensus.cc:504] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.908799 16703 ts_tablet_manager.cc:1398] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:15.909229 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a55a182e1f54db698ba8060eea80da0" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:15.909467 15754 raft_consensus.cc:3037] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.909552 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a55a182e1f54db698ba8060eea80da0" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:15.909622 16657 leader_election.cc:288] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:15.909726 12992 raft_consensus.cc:3037] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.910346 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a55a182e1f54db698ba8060eea80da0" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:15.910940 16619 leader_election.cc:288] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:15.911903 12992 raft_consensus.cc:2454] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:15.912140 15754 raft_consensus.cc:2454] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:15.912271 15419 leader_election.cc:302] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:15.912647 16619 raft_consensus.cc:2781] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:15.912761 16619 raft_consensus.cc:686] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:15.913142 16619 consensus_queue.cc:237] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:15.924580 12579 catalog_manager.cc:5484] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:16.164307 16580 raft_consensus.cc:482] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:16.164481 16580 raft_consensus.cc:504] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:16.165129 16580 leader_election.cc:288] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:16.165529 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ee60132711414b94627a2ebe4f2c0b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:16.165678 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ee60132711414b94627a2ebe4f2c0b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:16.165747 12859 raft_consensus.cc:2454] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:16.165935 15487 raft_consensus.cc:2454] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:16.166117 15554 leader_election.cc:302] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:16.166406 16580 raft_consensus.cc:2781] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:16.166558 16580 raft_consensus.cc:482] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:16.166656 16580 raft_consensus.cc:3037] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.168450 16580 raft_consensus.cc:504] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:16.168972 16580 leader_election.cc:288] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:16.169447 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ee60132711414b94627a2ebe4f2c0b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:16.169452 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ee60132711414b94627a2ebe4f2c0b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:16.169658 15487 raft_consensus.cc:3037] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.169735 12859 raft_consensus.cc:3037] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.171618 15487 raft_consensus.cc:2454] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:16.171864 12859 raft_consensus.cc:2454] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:16.172084 15554 leader_election.cc:302] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:16.172340 16580 raft_consensus.cc:2781] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:16.172498 16580 raft_consensus.cc:686] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:16.172983 16580 consensus_queue.cc:237] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:16.176174 12579 catalog_manager.cc:5484] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:16.207216 16619 consensus_queue.cc:1035] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.212818 16619 consensus_queue.cc:1035] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.290581 16657 consensus_queue.cc:1035] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.295848 16657 consensus_queue.cc:1035] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.299219 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:16.306186 12859 raft_consensus.cc:1261] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:16.306761 15488 raft_consensus.cc:1261] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:16.307081 16720 consensus_queue.cc:1035] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.307374 16709 consensus_queue.cc:1035] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.311833 16608 consensus_queue.cc:1035] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.312435 12579 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet a4ee60132711414b94627a2ebe4f2c0b (table rebalancer_dynamic_table [id=c2162eb2b6e041a6a20084fbfed02b99]). Expected version 1 got 0
I20240502 08:18:16.313367 16723 tablet.cc:1722] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:16.314215 16723 tablet.cc:1620] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:16.314954 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet a4ee60132711414b94627a2ebe4f2c0b (table rebalancer_dynamic_table [id=c2162eb2b6e041a6a20084fbfed02b99]). Expected version 1 got 0
I20240502 08:18:16.316447 16724 tablet.cc:1722] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:16.317368 16724 tablet.cc:1620] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:16.326380 16726 tablet.cc:1722] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:16.327317 16726 tablet.cc:1620] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:16.328187 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet a4ee60132711414b94627a2ebe4f2c0b (table rebalancer_dynamic_table [id=c2162eb2b6e041a6a20084fbfed02b99]). Expected version 1 got 0
I20240502 08:18:16.330309 12587 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet a4ee60132711414b94627a2ebe4f2c0b (table rebalancer_dynamic_table [id=c2162eb2b6e041a6a20084fbfed02b99]). Expected version 1 got 0
I20240502 08:18:16.332993 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=c2162eb2b6e041a6a20084fbfed02b99] alter complete (version 1)
I20240502 08:18:16.353255 16608 consensus_queue.cc:1035] T d6fdafe07a2846ebbea7a18f46c90794 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.440500 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:16.448832 16723 tablet.cc:1722] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:16.449316 16723 tablet.cc:1620] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:16.449469 16724 tablet.cc:1722] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:16.449891 16724 tablet.cc:1620] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:16.453953 16726 tablet.cc:1722] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:16.454397 16726 tablet.cc:1620] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:16.458354 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=c2162eb2b6e041a6a20084fbfed02b99] alter complete (version 2)
I20240502 08:18:16.503212 12587 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:16.512305 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 10ba9ba78b5444a4bc73be4f6cfd6370 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=efafa4d6445f46888867ab1040ffde69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.512544 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 10ba9ba78b5444a4bc73be4f6cfd6370 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=efafa4d6445f46888867ab1040ffde69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.512815 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ba9ba78b5444a4bc73be4f6cfd6370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.512941 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ba9ba78b5444a4bc73be4f6cfd6370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.513504 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 10ba9ba78b5444a4bc73be4f6cfd6370 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=efafa4d6445f46888867ab1040ffde69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.513861 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ba9ba78b5444a4bc73be4f6cfd6370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.519304 15867 tablet_service.cc:1467] Processing CreateTablet for tablet d0dede000b7943439f7b8455ea370e14 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=efafa4d6445f46888867ab1040ffde69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.520779 13102 tablet_service.cc:1467] Processing CreateTablet for tablet d0dede000b7943439f7b8455ea370e14 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=efafa4d6445f46888867ab1040ffde69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.521207 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0dede000b7943439f7b8455ea370e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.521234 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0dede000b7943439f7b8455ea370e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.521286 15600 tablet_service.cc:1467] Processing CreateTablet for tablet 84204315ebcf4ccbb0a62de8daa5cb23 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=efafa4d6445f46888867ab1040ffde69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.521636 15600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84204315ebcf4ccbb0a62de8daa5cb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.526615 16737 tablet_bootstrap.cc:492] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:16.526814 15599 tablet_service.cc:1467] Processing CreateTablet for tablet d0dede000b7943439f7b8455ea370e14 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=efafa4d6445f46888867ab1040ffde69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.527153 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0dede000b7943439f7b8455ea370e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.527655 16736 tablet_bootstrap.cc:492] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:16.528225 16737 tablet_bootstrap.cc:654] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.528888 16735 tablet_bootstrap.cc:492] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:16.529244 16736 tablet_bootstrap.cc:654] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.530543 16735 tablet_bootstrap.cc:654] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.536325 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 84204315ebcf4ccbb0a62de8daa5cb23 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=efafa4d6445f46888867ab1040ffde69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.536773 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84204315ebcf4ccbb0a62de8daa5cb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.537256 16736 tablet_bootstrap.cc:492] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:16.537400 16736 ts_tablet_manager.cc:1367] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 08:18:16.538086 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 84204315ebcf4ccbb0a62de8daa5cb23 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=efafa4d6445f46888867ab1040ffde69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.538551 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84204315ebcf4ccbb0a62de8daa5cb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.540977 16741 tablet_bootstrap.cc:492] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:16.543787 16741 tablet_bootstrap.cc:654] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.544804 16736 raft_consensus.cc:348] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.545070 16736 raft_consensus.cc:374] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.545218 16736 raft_consensus.cc:729] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.545529 16736 consensus_queue.cc:260] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.545768 16737 tablet_bootstrap.cc:492] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:16.546180 16736 ts_tablet_manager.cc:1398] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:18:16.546553 16736 tablet_bootstrap.cc:492] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:16.546865 16737 ts_tablet_manager.cc:1367] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.011s
I20240502 08:18:16.547747 16737 raft_consensus.cc:348] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.548059 16736 tablet_bootstrap.cc:654] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.548897 16737 raft_consensus.cc:374] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.549175 16737 raft_consensus.cc:729] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.550432 16737 consensus_queue.cc:260] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.551491 16742 tablet_bootstrap.cc:492] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:16.553110 16737 ts_tablet_manager.cc:1398] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:16.554277 16743 tablet_bootstrap.cc:492] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:16.555899 16743 tablet_bootstrap.cc:654] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.556118 16742 tablet_bootstrap.cc:654] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.557626 16735 tablet_bootstrap.cc:492] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:16.557853 16735 ts_tablet_manager.cc:1367] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.013s
I20240502 08:18:16.558679 16741 tablet_bootstrap.cc:492] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:16.558874 16741 ts_tablet_manager.cc:1367] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.006s
I20240502 08:18:16.558823 16735 raft_consensus.cc:348] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.559017 16735 raft_consensus.cc:374] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.559101 16735 raft_consensus.cc:729] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.559362 16735 consensus_queue.cc:260] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.559571 16741 raft_consensus.cc:348] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:16.559803 16741 raft_consensus.cc:374] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.559934 16741 raft_consensus.cc:729] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.560252 16741 consensus_queue.cc:260] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:16.560686 16735 ts_tablet_manager.cc:1398] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:16.561538 16735 tablet_bootstrap.cc:492] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:16.562387 16741 ts_tablet_manager.cc:1398] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:16.563362 16736 tablet_bootstrap.cc:492] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:16.563531 16736 ts_tablet_manager.cc:1367] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20240502 08:18:16.564386 16736 raft_consensus.cc:348] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:16.564608 16736 raft_consensus.cc:374] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.564671 16736 raft_consensus.cc:729] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.564895 16736 consensus_queue.cc:260] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:16.565169 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:16.565448 16736 ts_tablet_manager.cc:1398] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:16.565737 16736 tablet_bootstrap.cc:492] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:16.568850 16736 tablet_bootstrap.cc:654] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.569344 16743 tablet_bootstrap.cc:492] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:16.569533 16743 ts_tablet_manager.cc:1367] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 08:18:16.570371 16743 raft_consensus.cc:348] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:16.570636 16743 raft_consensus.cc:374] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.570780 16743 raft_consensus.cc:729] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.570852 16735 tablet_bootstrap.cc:654] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.571067 16743 consensus_queue.cc:260] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:16.573346 16743 ts_tablet_manager.cc:1398] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:16.577363 16723 tablet.cc:1722] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:16.580899 16736 tablet_bootstrap.cc:492] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:16.581043 16736 ts_tablet_manager.cc:1367] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20240502 08:18:16.581847 16736 raft_consensus.cc:348] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:16.582051 16736 raft_consensus.cc:374] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.582155 16736 raft_consensus.cc:729] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.582352 16736 consensus_queue.cc:260] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:16.582376 16724 tablet.cc:1722] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:16.582903 16736 ts_tablet_manager.cc:1398] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:16.586928 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=c2162eb2b6e041a6a20084fbfed02b99] alter complete (version 3)
I20240502 08:18:16.587390 16726 tablet.cc:1722] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:16.587960 16735 tablet_bootstrap.cc:492] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:16.588127 16735 ts_tablet_manager.cc:1367] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240502 08:18:16.588909 16735 raft_consensus.cc:348] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:16.589084 16735 raft_consensus.cc:374] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.589156 16735 raft_consensus.cc:729] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.589386 16735 consensus_queue.cc:260] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:16.589929 16735 ts_tablet_manager.cc:1398] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:16.590576 16709 raft_consensus.cc:482] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:16.590240 16742 tablet_bootstrap.cc:492] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:16.590735 16709 raft_consensus.cc:504] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:16.590971 16742 ts_tablet_manager.cc:1367] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.018s
I20240502 08:18:16.591395 16709 leader_election.cc:288] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:16.591998 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0dede000b7943439f7b8455ea370e14" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:16.592221 13121 raft_consensus.cc:2454] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:16.592602 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0dede000b7943439f7b8455ea370e14" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:16.592818 15886 raft_consensus.cc:2454] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:16.592933 15553 leader_election.cc:302] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:16.593873 16742 raft_consensus.cc:348] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:16.594014 16709 raft_consensus.cc:2781] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:16.594106 16709 raft_consensus.cc:482] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:16.594203 16709 raft_consensus.cc:3037] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.594830 16742 raft_consensus.cc:374] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.594987 16742 raft_consensus.cc:729] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.595501 16742 consensus_queue.cc:260] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:16.596026 16742 ts_tablet_manager.cc:1398] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:18:16.596642 16709 raft_consensus.cc:504] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:16.597267 16709 leader_election.cc:288] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:16.597704 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0dede000b7943439f7b8455ea370e14" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:16.597877 15886 raft_consensus.cc:3037] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.598306 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0dede000b7943439f7b8455ea370e14" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:16.598503 13121 raft_consensus.cc:3037] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.600028 15886 raft_consensus.cc:2454] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:16.600553 15555 leader_election.cc:302] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:16.600849 16709 raft_consensus.cc:2781] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:16.601024 16709 raft_consensus.cc:686] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:16.601258 13121 raft_consensus.cc:2454] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:16.601336 16709 consensus_queue.cc:237] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:16.606406 12586 catalog_manager.cc:5484] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:16.621919 16709 raft_consensus.cc:482] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:16.622092 16709 raft_consensus.cc:504] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.622743 16709 leader_election.cc:288] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:16.623167 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ba9ba78b5444a4bc73be4f6cfd6370" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:16.623199 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ba9ba78b5444a4bc73be4f6cfd6370" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:16.623378 15754 raft_consensus.cc:2454] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:16.623430 15886 raft_consensus.cc:2454] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:16.623842 15555 leader_election.cc:302] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:16.624102 16709 raft_consensus.cc:2781] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:16.624255 16709 raft_consensus.cc:482] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:16.624343 16709 raft_consensus.cc:3037] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.627328 16709 raft_consensus.cc:504] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.627980 16709 leader_election.cc:288] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:16.628391 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ba9ba78b5444a4bc73be4f6cfd6370" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:16.628589 15886 raft_consensus.cc:3037] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.631295 15886 raft_consensus.cc:2454] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:16.631637 15555 leader_election.cc:302] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:16.631937 16709 raft_consensus.cc:2781] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:16.632056 16709 raft_consensus.cc:686] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:16.632297 16709 consensus_queue.cc:237] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.635699 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ba9ba78b5444a4bc73be4f6cfd6370" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:16.635886 15754 raft_consensus.cc:3037] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.635998 12586 catalog_manager.cc:5484] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:16.638118 15754 raft_consensus.cc:2454] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:16.694350 12586 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:16.694590 12586 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:16.697286 12586 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet a4ee60132711414b94627a2ebe4f2c0b
I20240502 08:18:16.698019 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet a4ee60132711414b94627a2ebe4f2c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:16 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:16.698186 15601 tablet_service.cc:1514] Processing DeleteTablet for tablet a4ee60132711414b94627a2ebe4f2c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:16 UTC) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:16.698233 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet a4ee60132711414b94627a2ebe4f2c0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:16 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:16.698731 16753 tablet_replica.cc:330] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:16.699036 16753 raft_consensus.cc:2229] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:16.699327 16753 raft_consensus.cc:2258] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.700111 16755 tablet_replica.cc:330] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:16.700340 16754 tablet_replica.cc:330] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:16.700430 16755 raft_consensus.cc:2229] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:16.700584 16754 raft_consensus.cc:2229] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:16.700651 16755 raft_consensus.cc:2258] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.701105 16753 ts_tablet_manager.cc:1875] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:16.701360 16754 raft_consensus.cc:2258] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:16.701489 16755 ts_tablet_manager.cc:1875] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:16.702169 16754 ts_tablet_manager.cc:1875] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:16.706218 16753 ts_tablet_manager.cc:1888] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:16.706337 16753 log.cc:1196] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/a4ee60132711414b94627a2ebe4f2c0b
I20240502 08:18:16.706893 16753 ts_tablet_manager.cc:1909] T a4ee60132711414b94627a2ebe4f2c0b P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:16.707571 16755 ts_tablet_manager.cc:1888] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:16.707695 16755 log.cc:1196] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/a4ee60132711414b94627a2ebe4f2c0b
I20240502 08:18:16.707949 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet a4ee60132711414b94627a2ebe4f2c0b (table rebalancer_dynamic_table_new_name [id=c2162eb2b6e041a6a20084fbfed02b99]) successfully deleted
I20240502 08:18:16.708202 16755 ts_tablet_manager.cc:1909] T a4ee60132711414b94627a2ebe4f2c0b P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:16.708745 16754 ts_tablet_manager.cc:1888] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:16.708868 16754 log.cc:1196] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/a4ee60132711414b94627a2ebe4f2c0b
I20240502 08:18:16.709081 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet a4ee60132711414b94627a2ebe4f2c0b (table rebalancer_dynamic_table_new_name [id=c2162eb2b6e041a6a20084fbfed02b99]) successfully deleted
I20240502 08:18:16.709260 16754 ts_tablet_manager.cc:1909] T a4ee60132711414b94627a2ebe4f2c0b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting consensus metadata
I20240502 08:18:16.710018 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet a4ee60132711414b94627a2ebe4f2c0b (table rebalancer_dynamic_table_new_name [id=c2162eb2b6e041a6a20084fbfed02b99]) successfully deleted
I20240502 08:18:16.799906 12586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:16.807467 15734 tablet_service.cc:1467] Processing CreateTablet for tablet ef93d5d589ad414c9aeb71dd85472d1a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59ef1021b86f41c6a88b6365c2731b07]), partition=
I20240502 08:18:16.807860 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef93d5d589ad414c9aeb71dd85472d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.807838 15468 tablet_service.cc:1467] Processing CreateTablet for tablet ef93d5d589ad414c9aeb71dd85472d1a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59ef1021b86f41c6a88b6365c2731b07]), partition=
I20240502 08:18:16.808116 15865 tablet_service.cc:1467] Processing CreateTablet for tablet ef93d5d589ad414c9aeb71dd85472d1a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59ef1021b86f41c6a88b6365c2731b07]), partition=
I20240502 08:18:16.808202 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef93d5d589ad414c9aeb71dd85472d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.808490 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef93d5d589ad414c9aeb71dd85472d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.814545 16735 tablet_bootstrap.cc:492] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:16.814766 16742 tablet_bootstrap.cc:492] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:16.815941 16735 tablet_bootstrap.cc:654] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.816146 16742 tablet_bootstrap.cc:654] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.816385 16737 tablet_bootstrap.cc:492] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:16.817878 16737 tablet_bootstrap.cc:654] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.825284 16737 tablet_bootstrap.cc:492] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:16.825435 16737 ts_tablet_manager.cc:1367] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240502 08:18:16.825672 16735 tablet_bootstrap.cc:492] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:16.825790 16735 ts_tablet_manager.cc:1367] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 08:18:16.826007 16742 tablet_bootstrap.cc:492] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:16.826138 16742 ts_tablet_manager.cc:1367] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20240502 08:18:16.826134 16737 raft_consensus.cc:348] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.826347 16737 raft_consensus.cc:374] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.826445 16735 raft_consensus.cc:348] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.826417 16737 raft_consensus.cc:729] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.826689 16735 raft_consensus.cc:374] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.826814 16735 raft_consensus.cc:729] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.826797 16737 consensus_queue.cc:260] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.826822 16742 raft_consensus.cc:348] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.827010 16742 raft_consensus.cc:374] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.827062 16742 raft_consensus.cc:729] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.827077 16735 consensus_queue.cc:260] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.827296 16737 ts_tablet_manager.cc:1398] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:16.827286 16742 consensus_queue.cc:260] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.827877 16742 ts_tablet_manager.cc:1398] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:16.831708 16735 ts_tablet_manager.cc:1398] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 08:18:16.836278 16728 raft_consensus.cc:482] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:16.836421 16728 raft_consensus.cc:504] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.837944 16728 leader_election.cc:288] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:16.838418 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef93d5d589ad414c9aeb71dd85472d1a" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:16.838547 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef93d5d589ad414c9aeb71dd85472d1a" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:16.838670 15487 raft_consensus.cc:2454] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:16.838747 15754 raft_consensus.cc:2454] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:16.839092 15819 leader_election.cc:302] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:16.839532 16728 raft_consensus.cc:2781] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:16.839632 16728 raft_consensus.cc:482] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:16.839715 16728 raft_consensus.cc:3037] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.842222 16728 raft_consensus.cc:504] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.843065 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef93d5d589ad414c9aeb71dd85472d1a" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:16.843247 15754 raft_consensus.cc:3037] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.843935 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef93d5d589ad414c9aeb71dd85472d1a" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:16.844110 15487 raft_consensus.cc:3037] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.844671 16728 leader_election.cc:288] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:16.845609 15754 raft_consensus.cc:2454] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:16.845942 15818 leader_election.cc:302] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:16.846179 16728 raft_consensus.cc:2781] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:16.846197 15487 raft_consensus.cc:2454] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:16.846297 16728 raft_consensus.cc:686] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:16.846565 16728 consensus_queue.cc:237] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:16.849524 12587 catalog_manager.cc:5484] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:16.861440 16709 raft_consensus.cc:482] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:16.861588 16709 raft_consensus.cc:504] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:16.862143 16709 leader_election.cc:288] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:16.862605 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204315ebcf4ccbb0a62de8daa5cb23" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:16.862692 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204315ebcf4ccbb0a62de8daa5cb23" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:16.862766 15487 raft_consensus.cc:2454] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:16.863020 12992 raft_consensus.cc:2454] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:16.863163 15554 leader_election.cc:302] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:16.863432 16709 raft_consensus.cc:2781] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:16.863538 16709 raft_consensus.cc:482] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:16.863638 16709 raft_consensus.cc:3037] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.865705 16709 raft_consensus.cc:504] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:16.866189 16709 leader_election.cc:288] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:16.866631 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204315ebcf4ccbb0a62de8daa5cb23" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:16.866619 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204315ebcf4ccbb0a62de8daa5cb23" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:16.866771 15487 raft_consensus.cc:3037] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.866801 12992 raft_consensus.cc:3037] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.868593 12992 raft_consensus.cc:2454] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:16.868646 15487 raft_consensus.cc:2454] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:16.868916 15554 leader_election.cc:302] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:16.869138 16709 raft_consensus.cc:2781] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:16.869323 16709 raft_consensus.cc:686] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:16.869547 16709 consensus_queue.cc:237] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:16.872116 12587 catalog_manager.cc:5484] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:16.955368 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:16.961412 15487 raft_consensus.cc:1261] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:16.961648 15754 raft_consensus.cc:1261] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:16.962177 16632 consensus_queue.cc:1035] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.962430 16728 consensus_queue.cc:1035] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.966746 12586 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet ef93d5d589ad414c9aeb71dd85472d1a (table rebalancer_dynamic_table [id=59ef1021b86f41c6a88b6365c2731b07]). Expected version 1 got 0
I20240502 08:18:16.967725 16766 tablet.cc:1722] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:16.968425 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet ef93d5d589ad414c9aeb71dd85472d1a (table rebalancer_dynamic_table [id=59ef1021b86f41c6a88b6365c2731b07]). Expected version 1 got 0
I20240502 08:18:16.968495 16766 tablet.cc:1620] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:16.970391 16724 tablet.cc:1722] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:16.970896 16724 tablet.cc:1620] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:16.970230 16768 tablet.cc:1722] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:16.971851 16768 tablet.cc:1620] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:16.973580 12587 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet ef93d5d589ad414c9aeb71dd85472d1a (table rebalancer_dynamic_table [id=59ef1021b86f41c6a88b6365c2731b07]). Expected version 1 got 0
I20240502 08:18:16.975627 12584 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet ef93d5d589ad414c9aeb71dd85472d1a (table rebalancer_dynamic_table [id=59ef1021b86f41c6a88b6365c2731b07]). Expected version 1 got 0
I20240502 08:18:16.982005 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=59ef1021b86f41c6a88b6365c2731b07] alter complete (version 1)
I20240502 08:18:17.085638 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:17.092172 16766 tablet.cc:1722] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:17.092638 16766 tablet.cc:1620] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:17.092624 16724 tablet.cc:1722] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:17.093094 16724 tablet.cc:1620] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:17.093973 16768 tablet.cc:1722] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:17.094328 16768 tablet.cc:1620] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:17.102126 16709 consensus_queue.cc:1035] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.104509 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=59ef1021b86f41c6a88b6365c2731b07] alter complete (version 2)
I20240502 08:18:17.106666 16709 consensus_queue.cc:1035] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.121482 16580 consensus_queue.cc:1035] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.125180 16709 consensus_queue.cc:1035] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.204423 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:17.212307 16766 tablet.cc:1722] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:17.213343 16724 tablet.cc:1722] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:17.214579 16768 tablet.cc:1722] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:17.221664 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=59ef1021b86f41c6a88b6365c2731b07] alter complete (version 3)
I20240502 08:18:17.310015 16709 consensus_queue.cc:1035] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.315095 16709 consensus_queue.cc:1035] T 84204315ebcf4ccbb0a62de8daa5cb23 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.325676 12579 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:17.325884 12579 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:17.329021 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet ef93d5d589ad414c9aeb71dd85472d1a
I20240502 08:18:17.329916 15734 tablet_service.cc:1514] Processing DeleteTablet for tablet ef93d5d589ad414c9aeb71dd85472d1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:17 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:17.329957 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet ef93d5d589ad414c9aeb71dd85472d1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:17 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:17.330173 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet ef93d5d589ad414c9aeb71dd85472d1a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:17 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:17.332760 16783 tablet_replica.cc:330] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:17.332602 16782 tablet_replica.cc:330] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:17.333038 16783 raft_consensus.cc:2229] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:17.333267 16782 raft_consensus.cc:2229] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:17.333516 16782 raft_consensus.cc:2258] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.333575 16783 raft_consensus.cc:2258] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.334431 16781 tablet_replica.cc:330] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:17.334612 16782 ts_tablet_manager.cc:1875] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:17.334638 16781 raft_consensus.cc:2229] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:17.334831 16781 raft_consensus.cc:2258] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:17.335017 16783 ts_tablet_manager.cc:1875] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:17.335669 16781 ts_tablet_manager.cc:1875] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:17.341579 16781 ts_tablet_manager.cc:1888] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:17.341758 16781 log.cc:1196] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/ef93d5d589ad414c9aeb71dd85472d1a
I20240502 08:18:17.341784 16782 ts_tablet_manager.cc:1888] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:17.341919 16783 ts_tablet_manager.cc:1888] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:17.341989 16782 log.cc:1196] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/ef93d5d589ad414c9aeb71dd85472d1a
I20240502 08:18:17.342084 16783 log.cc:1196] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/ef93d5d589ad414c9aeb71dd85472d1a
I20240502 08:18:17.342288 16781 ts_tablet_manager.cc:1909] T ef93d5d589ad414c9aeb71dd85472d1a P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:17.342543 16782 ts_tablet_manager.cc:1909] T ef93d5d589ad414c9aeb71dd85472d1a P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:17.342561 16783 ts_tablet_manager.cc:1909] T ef93d5d589ad414c9aeb71dd85472d1a P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:17.343343 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet ef93d5d589ad414c9aeb71dd85472d1a (table rebalancer_dynamic_table_new_name [id=59ef1021b86f41c6a88b6365c2731b07]) successfully deleted
I20240502 08:18:17.343494 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet ef93d5d589ad414c9aeb71dd85472d1a (table rebalancer_dynamic_table_new_name [id=59ef1021b86f41c6a88b6365c2731b07]) successfully deleted
I20240502 08:18:17.343845 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet ef93d5d589ad414c9aeb71dd85472d1a (table rebalancer_dynamic_table_new_name [id=59ef1021b86f41c6a88b6365c2731b07]) successfully deleted
I20240502 08:18:17.412228 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:17.420540 15865 tablet_service.cc:1467] Processing CreateTablet for tablet ead2e2b9f05a4a729bcba36dcec35a0a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=602cdebb879f43d3b1a9666d8bd7cddc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.420894 15468 tablet_service.cc:1467] Processing CreateTablet for tablet ead2e2b9f05a4a729bcba36dcec35a0a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=602cdebb879f43d3b1a9666d8bd7cddc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.421006 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead2e2b9f05a4a729bcba36dcec35a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.421232 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead2e2b9f05a4a729bcba36dcec35a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.421314 15467 tablet_service.cc:1467] Processing CreateTablet for tablet c60b24cf2d4740ae9b8829e288bc2530 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=602cdebb879f43d3b1a9666d8bd7cddc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.421667 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60b24cf2d4740ae9b8829e288bc2530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.423135 13102 tablet_service.cc:1467] Processing CreateTablet for tablet ead2e2b9f05a4a729bcba36dcec35a0a (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=602cdebb879f43d3b1a9666d8bd7cddc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.423453 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead2e2b9f05a4a729bcba36dcec35a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.429100 16784 tablet_bootstrap.cc:492] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:17.430800 16784 tablet_bootstrap.cc:654] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.432078 15867 tablet_service.cc:1467] Processing CreateTablet for tablet c60b24cf2d4740ae9b8829e288bc2530 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=602cdebb879f43d3b1a9666d8bd7cddc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.432490 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60b24cf2d4740ae9b8829e288bc2530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.432559 15734 tablet_service.cc:1467] Processing CreateTablet for tablet c60b24cf2d4740ae9b8829e288bc2530 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=602cdebb879f43d3b1a9666d8bd7cddc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.432946 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60b24cf2d4740ae9b8829e288bc2530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.433291 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:17.437142 15866 tablet_service.cc:1467] Processing CreateTablet for tablet a93ecff854b044349a8cdcee14fba2cb (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=602cdebb879f43d3b1a9666d8bd7cddc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.437665 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93ecff854b044349a8cdcee14fba2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.439744 12709 tablet_service.cc:1467] Processing CreateTablet for tablet a93ecff854b044349a8cdcee14fba2cb (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=602cdebb879f43d3b1a9666d8bd7cddc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.440135 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93ecff854b044349a8cdcee14fba2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.441293 15467 tablet_service.cc:1467] Processing CreateTablet for tablet a93ecff854b044349a8cdcee14fba2cb (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=602cdebb879f43d3b1a9666d8bd7cddc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:17.441625 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93ecff854b044349a8cdcee14fba2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.450613 16784 tablet_bootstrap.cc:492] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:17.450811 16784 ts_tablet_manager.cc:1367] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.004s
I20240502 08:18:17.451119 16786 tablet_bootstrap.cc:492] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:17.451567 16784 raft_consensus.cc:348] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:17.451804 16784 raft_consensus.cc:374] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.451934 16784 raft_consensus.cc:729] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.452236 16784 consensus_queue.cc:260] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:17.452831 16784 ts_tablet_manager.cc:1398] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:17.453411 16784 tablet_bootstrap.cc:492] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:17.454026 16786 tablet_bootstrap.cc:654] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.454850 16784 tablet_bootstrap.cc:654] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.455587 16789 tablet_bootstrap.cc:492] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:17.457295 16789 tablet_bootstrap.cc:654] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.460634 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 2e128c52597744d29781a27c862a16a6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f11046a7e0184e7e8ad174f7ade96dc9]), partition=
I20240502 08:18:17.460965 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e128c52597744d29781a27c862a16a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.463133 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 2e128c52597744d29781a27c862a16a6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f11046a7e0184e7e8ad174f7ade96dc9]), partition=
I20240502 08:18:17.463503 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e128c52597744d29781a27c862a16a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.465579 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 2e128c52597744d29781a27c862a16a6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f11046a7e0184e7e8ad174f7ade96dc9]), partition=
I20240502 08:18:17.465888 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e128c52597744d29781a27c862a16a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:17.476711 16789 tablet_bootstrap.cc:492] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:17.477293 16789 ts_tablet_manager.cc:1367] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.004s
I20240502 08:18:17.478364 16789 raft_consensus.cc:348] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.479692 16789 raft_consensus.cc:374] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.479934 16789 raft_consensus.cc:729] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.480293 16793 tablet_bootstrap.cc:492] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:17.480320 16789 consensus_queue.cc:260] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.482734 16789 ts_tablet_manager.cc:1398] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:17.483309 16789 tablet_bootstrap.cc:492] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:17.483548 16793 tablet_bootstrap.cc:654] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.485319 16787 tablet_bootstrap.cc:492] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:17.486268 16789 tablet_bootstrap.cc:654] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.490998 16787 tablet_bootstrap.cc:654] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.492921 16788 tablet_bootstrap.cc:492] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:17.495033 16784 tablet_bootstrap.cc:492] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:17.495215 16793 tablet_bootstrap.cc:492] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:17.495285 16784 ts_tablet_manager.cc:1367] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.015s
I20240502 08:18:17.495386 16793 ts_tablet_manager.cc:1367] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 08:18:17.496052 16784 raft_consensus.cc:348] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.496138 16793 raft_consensus.cc:348] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.496342 16784 raft_consensus.cc:374] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.496413 16784 raft_consensus.cc:729] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.496484 16793 raft_consensus.cc:374] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.496604 16793 raft_consensus.cc:729] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.496719 16784 consensus_queue.cc:260] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.496891 16793 consensus_queue.cc:260] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.497529 16793 ts_tablet_manager.cc:1398] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:17.497917 16788 tablet_bootstrap.cc:654] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.498327 16784 ts_tablet_manager.cc:1398] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:17.498678 16784 tablet_bootstrap.cc:492] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:17.501698 16789 tablet_bootstrap.cc:492] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:17.502100 16784 tablet_bootstrap.cc:654] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.502382 16786 tablet_bootstrap.cc:492] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:17.502614 16786 ts_tablet_manager.cc:1367] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.012s
I20240502 08:18:17.503652 16786 raft_consensus.cc:348] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:17.503960 16786 raft_consensus.cc:374] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.504099 16786 raft_consensus.cc:729] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.504397 16786 consensus_queue.cc:260] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:17.507853 16786 ts_tablet_manager.cc:1398] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 08:18:17.510955 16786 tablet_bootstrap.cc:492] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:17.511451 16789 ts_tablet_manager.cc:1367] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.005s
I20240502 08:18:17.511714 16787 tablet_bootstrap.cc:492] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:17.511924 16787 ts_tablet_manager.cc:1367] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240502 08:18:17.512800 16787 raft_consensus.cc:348] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:17.513067 16787 raft_consensus.cc:374] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.513290 16787 raft_consensus.cc:729] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.513422 16789 raft_consensus.cc:348] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.513581 16787 consensus_queue.cc:260] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:17.515161 16787 ts_tablet_manager.cc:1398] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:17.515540 16787 tablet_bootstrap.cc:492] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:17.515589 16789 raft_consensus.cc:374] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.515738 16786 tablet_bootstrap.cc:654] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.516631 16789 raft_consensus.cc:729] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.516968 16789 consensus_queue.cc:260] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.517369 16777 raft_consensus.cc:482] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.517508 16777 raft_consensus.cc:504] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.518146 16787 tablet_bootstrap.cc:654] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.518169 16777 leader_election.cc:288] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:17.518808 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ecff854b044349a8cdcee14fba2cb" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:17.519712 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ecff854b044349a8cdcee14fba2cb" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:17.519915 12730 raft_consensus.cc:2454] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:17.520463 15419 leader_election.cc:302] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
W20240502 08:18:17.520961 15420 leader_election.cc:341] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:17.521294 16789 ts_tablet_manager.cc:1398] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 08:18:17.521675 16777 raft_consensus.cc:2781] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:17.521785 16777 raft_consensus.cc:482] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:17.521875 16777 raft_consensus.cc:3037] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.524515 16777 raft_consensus.cc:504] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.526928 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ecff854b044349a8cdcee14fba2cb" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:17.527118 12730 raft_consensus.cc:3037] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.529783 12730 raft_consensus.cc:2454] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:17.530241 16788 tablet_bootstrap.cc:492] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:17.530417 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ecff854b044349a8cdcee14fba2cb" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
W20240502 08:18:17.530937 15420 leader_election.cc:341] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:17.531239 16788 ts_tablet_manager.cc:1367] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.010s
I20240502 08:18:17.531580 15419 leader_election.cc:302] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters: 10f0e36203914b85b77e78609ffced98
I20240502 08:18:17.531844 16619 raft_consensus.cc:2781] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:17.532761 16787 tablet_bootstrap.cc:492] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:17.532966 16787 ts_tablet_manager.cc:1367] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.004s
I20240502 08:18:17.533046 16777 leader_election.cc:288] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:17.533490 16759 raft_consensus.cc:482] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.533464 16788 raft_consensus.cc:348] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:17.533654 16788 raft_consensus.cc:374] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.533607 16759 raft_consensus.cc:504] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.533735 16788 raft_consensus.cc:729] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.533766 16787 raft_consensus.cc:348] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:17.533960 16788 consensus_queue.cc:260] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:17.534066 16787 raft_consensus.cc:374] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.534224 16787 raft_consensus.cc:729] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.534369 16759 leader_election.cc:288] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:17.534529 16787 consensus_queue.cc:260] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:17.535108 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e128c52597744d29781a27c862a16a6" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:17.535310 12859 raft_consensus.cc:2454] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:17.535638 16788 ts_tablet_manager.cc:1398] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:18:17.535838 16787 ts_tablet_manager.cc:1398] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:17.536115 16787 tablet_bootstrap.cc:492] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:17.536511 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e128c52597744d29781a27c862a16a6" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:17.537050 12664 leader_election.cc:302] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
W20240502 08:18:17.537501 12663 leader_election.cc:341] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:17.537815 16707 raft_consensus.cc:2781] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:17.537912 16707 raft_consensus.cc:482] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:17.538002 16707 raft_consensus.cc:3037] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.538641 16787 tablet_bootstrap.cc:654] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:17.540534 16707 raft_consensus.cc:504] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.541891 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e128c52597744d29781a27c862a16a6" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:17.542071 12859 raft_consensus.cc:3037] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.543020 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e128c52597744d29781a27c862a16a6" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:17.543658 16707 leader_election.cc:288] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
W20240502 08:18:17.543956 12663 leader_election.cc:341] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:17.544324 16619 raft_consensus.cc:686] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:17.544611 16619 consensus_queue.cc:237] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.546022 12859 raft_consensus.cc:2454] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:17.546437 12664 leader_election.cc:302] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters: 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:17.548058 12584 catalog_manager.cc:5484] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:17.549162 16786 tablet_bootstrap.cc:492] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:17.549441 16786 ts_tablet_manager.cc:1367] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.003s
I20240502 08:18:17.549561 16759 raft_consensus.cc:2781] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:17.549714 16759 raft_consensus.cc:686] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:18:17.549952 16759 consensus_queue.cc:237] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.550222 16786 raft_consensus.cc:348] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.550436 16786 raft_consensus.cc:374] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.550601 16786 raft_consensus.cc:729] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.550928 16786 consensus_queue.cc:260] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.551638 16767 raft_consensus.cc:482] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.551740 16767 raft_consensus.cc:504] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:17.552263 16767 leader_election.cc:288] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:17.552834 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60b24cf2d4740ae9b8829e288bc2530" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:17.553056 15487 raft_consensus.cc:2454] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:17.553680 15819 leader_election.cc:302] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:17.554260 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60b24cf2d4740ae9b8829e288bc2530" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:17.554438 15754 raft_consensus.cc:2454] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:17.554595 16767 raft_consensus.cc:2781] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:17.554694 16767 raft_consensus.cc:482] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:17.554781 16767 raft_consensus.cc:3037] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.555580 16786 ts_tablet_manager.cc:1398] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:17.556047 16784 tablet_bootstrap.cc:492] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:17.556185 16784 ts_tablet_manager.cc:1367] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.010s
I20240502 08:18:17.556186 16477 raft_consensus.cc:482] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.556326 16477 raft_consensus.cc:504] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.556898 16767 raft_consensus.cc:504] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:17.556941 16784 raft_consensus.cc:348] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:17.557072 16477 leader_election.cc:288] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:17.557109 16784 raft_consensus.cc:374] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.557193 16784 raft_consensus.cc:729] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.557468 16767 leader_election.cc:288] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:17.557476 16784 consensus_queue.cc:260] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:17.557823 16787 tablet_bootstrap.cc:492] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:17.557943 16787 ts_tablet_manager.cc:1367] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20240502 08:18:17.557966 16784 ts_tablet_manager.cc:1398] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:17.558538 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e128c52597744d29781a27c862a16a6" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:17.558683 16787 raft_consensus.cc:348] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.558838 16787 raft_consensus.cc:374] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:17.558919 16787 raft_consensus.cc:729] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:17.559095 16787 consensus_queue.cc:260] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.559593 16787 ts_tablet_manager.cc:1398] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:17.561232 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60b24cf2d4740ae9b8829e288bc2530" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:17.561429 15487 raft_consensus.cc:3037] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.564074 15487 raft_consensus.cc:2454] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:17.564810 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60b24cf2d4740ae9b8829e288bc2530" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:17.565006 15754 raft_consensus.cc:3037] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.567091 15754 raft_consensus.cc:2454] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:17.567423 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e128c52597744d29781a27c862a16a6" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:17.567467 15819 leader_election.cc:302] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:17.567698 12859 raft_consensus.cc:2379] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9333cb10227b4801b4fa7c24e22e131b in current term 1: Already voted for candidate 29b035434087435cbaf4ebcac98eedee in this term.
I20240502 08:18:17.567878 16632 raft_consensus.cc:2781] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:17.568001 16632 raft_consensus.cc:686] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:17.568233 15688 leader_election.cc:302] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b; no voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a
I20240502 08:18:17.568280 16632 consensus_queue.cc:237] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:17.568677 16477 raft_consensus.cc:3037] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.570140 12579 catalog_manager.cc:5484] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:17.570889 16477 raft_consensus.cc:2726] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:17.572718 12587 catalog_manager.cc:5484] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:17.686749 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:17.692420 12859 raft_consensus.cc:1261] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:17.692644 15754 raft_consensus.cc:1261] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:17.693054 16707 consensus_queue.cc:1035] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.693336 16809 consensus_queue.cc:1035] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.697343 12587 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet 2e128c52597744d29781a27c862a16a6 (table rebalancer_dynamic_table [id=f11046a7e0184e7e8ad174f7ade96dc9]). Expected version 1 got 0
I20240502 08:18:17.697458 16813 tablet.cc:1722] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:17.698129 16813 tablet.cc:1620] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:17.700641 16814 tablet.cc:1722] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:17.701529 16814 tablet.cc:1620] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:17.701637 12587 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet 2e128c52597744d29781a27c862a16a6 (table rebalancer_dynamic_table [id=f11046a7e0184e7e8ad174f7ade96dc9]). Expected version 1 got 0
I20240502 08:18:17.708686 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=f11046a7e0184e7e8ad174f7ade96dc9] alter complete (version 1)
I20240502 08:18:17.711385 16768 tablet.cc:1722] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:17.711843 16768 tablet.cc:1620] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:17.711232 12587 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet 2e128c52597744d29781a27c862a16a6 (table rebalancer_dynamic_table [id=f11046a7e0184e7e8ad174f7ade96dc9]). Expected version 1 got 0
I20240502 08:18:17.784907 16619 raft_consensus.cc:482] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.785068 16619 raft_consensus.cc:504] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:17.785643 16619 leader_election.cc:288] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:17.786067 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead2e2b9f05a4a729bcba36dcec35a0a" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:17.786082 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead2e2b9f05a4a729bcba36dcec35a0a" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:17.786294 15887 raft_consensus.cc:2454] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:17.786295 13119 raft_consensus.cc:2454] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:17.786700 15418 leader_election.cc:302] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:17.786943 16619 raft_consensus.cc:2781] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:17.787055 16619 raft_consensus.cc:482] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:17.787143 16619 raft_consensus.cc:3037] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.788903 16619 raft_consensus.cc:504] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:17.789423 16619 leader_election.cc:288] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:17.789768 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead2e2b9f05a4a729bcba36dcec35a0a" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:17.789820 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead2e2b9f05a4a729bcba36dcec35a0a" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:17.789959 15886 raft_consensus.cc:3037] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.789969 13119 raft_consensus.cc:3037] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.791927 13119 raft_consensus.cc:2454] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:17.792058 15886 raft_consensus.cc:2454] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:17.792254 15418 leader_election.cc:302] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:17.792526 16619 raft_consensus.cc:2781] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:17.792672 16619 raft_consensus.cc:686] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:17.792977 16619 consensus_queue.cc:237] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:17.796068 12587 catalog_manager.cc:5484] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:17.808230 16767 raft_consensus.cc:482] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.808369 16767 raft_consensus.cc:504] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:17.808964 16767 leader_election.cc:288] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:17.809412 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ecff854b044349a8cdcee14fba2cb" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:17.809490 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93ecff854b044349a8cdcee14fba2cb" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:17.809737 12730 raft_consensus.cc:2379] T a93ecff854b044349a8cdcee14fba2cb P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10f0e36203914b85b77e78609ffced98 in current term 1: Already voted for candidate db7893bb29a047b3b82cc60bfd8930a3 in this term.
I20240502 08:18:17.810257 15819 leader_election.cc:302] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10f0e36203914b85b77e78609ffced98; no voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:17.810511 16767 raft_consensus.cc:3037] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.812780 16767 raft_consensus.cc:2726] T a93ecff854b044349a8cdcee14fba2cb P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:17.813835 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:17.821331 16813 tablet.cc:1722] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:17.821839 16813 tablet.cc:1620] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:17.824251 16768 tablet.cc:1722] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:17.824738 16768 tablet.cc:1620] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:17.826364 16814 tablet.cc:1722] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:17.826856 16814 tablet.cc:1620] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:17.834044 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=f11046a7e0184e7e8ad174f7ade96dc9] alter complete (version 2)
I20240502 08:18:17.936697 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:17.943332 16813 tablet.cc:1722] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:17.944036 16814 tablet.cc:1722] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:17.944427 16768 tablet.cc:1722] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:17.952991 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=f11046a7e0184e7e8ad174f7ade96dc9] alter complete (version 3)
I20240502 08:18:18.008064 16767 consensus_queue.cc:1035] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.013656 16632 consensus_queue.cc:1035] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.022265 16619 consensus_queue.cc:1035] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 08:18:18.028928 16619 consensus_queue.cc:1035] T a93ecff854b044349a8cdcee14fba2cb P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.055738 12587 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:18.055945 12587 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:18.058604 12587 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 2e128c52597744d29781a27c862a16a6
I20240502 08:18:18.059449 15734 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e128c52597744d29781a27c862a16a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:18 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:18.059461 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e128c52597744d29781a27c862a16a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:18 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:18.059465 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e128c52597744d29781a27c862a16a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:18 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:18.062021 16825 tablet_replica.cc:330] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:18.062247 16825 raft_consensus.cc:2229] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:18.062485 16825 raft_consensus.cc:2258] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:18.063313 16825 ts_tablet_manager.cc:1875] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:18.068501 16825 ts_tablet_manager.cc:1888] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:18.068631 16825 log.cc:1196] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/2e128c52597744d29781a27c862a16a6
I20240502 08:18:18.069062 16825 ts_tablet_manager.cc:1909] T 2e128c52597744d29781a27c862a16a6 P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:18.069990 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 2e128c52597744d29781a27c862a16a6 (table rebalancer_dynamic_table_new_name [id=f11046a7e0184e7e8ad174f7ade96dc9]) successfully deleted
I20240502 08:18:18.070220 16827 tablet_replica.cc:330] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:18.070417 16827 raft_consensus.cc:2229] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:18.070804 16827 raft_consensus.cc:2258] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:18.071238 16826 tablet_replica.cc:330] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:18.071442 16827 ts_tablet_manager.cc:1875] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:18.071447 16826 raft_consensus.cc:2229] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:18.071661 16826 raft_consensus.cc:2258] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:18.072310 16826 ts_tablet_manager.cc:1875] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:18.075438 16827 ts_tablet_manager.cc:1888] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:18.075532 16827 log.cc:1196] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/2e128c52597744d29781a27c862a16a6
I20240502 08:18:18.075851 16827 ts_tablet_manager.cc:1909] T 2e128c52597744d29781a27c862a16a6 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:18.076426 16826 ts_tablet_manager.cc:1888] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:18.076555 16826 log.cc:1196] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/2e128c52597744d29781a27c862a16a6
I20240502 08:18:18.076694 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 2e128c52597744d29781a27c862a16a6 (table rebalancer_dynamic_table_new_name [id=f11046a7e0184e7e8ad174f7ade96dc9]) successfully deleted
I20240502 08:18:18.076952 16826 ts_tablet_manager.cc:1909] T 2e128c52597744d29781a27c862a16a6 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:18.077766 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 2e128c52597744d29781a27c862a16a6 (table rebalancer_dynamic_table_new_name [id=f11046a7e0184e7e8ad174f7ade96dc9]) successfully deleted
I20240502 08:18:18.161334 12587 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:18.169404 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 7d88c3e0a76347e5bac9e5d13b2da02c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f505f286631a443e9dfb99f883bfd9bb]), partition=
I20240502 08:18:18.169476 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 7d88c3e0a76347e5bac9e5d13b2da02c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f505f286631a443e9dfb99f883bfd9bb]), partition=
I20240502 08:18:18.169798 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d88c3e0a76347e5bac9e5d13b2da02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.169828 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d88c3e0a76347e5bac9e5d13b2da02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.174832 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 7d88c3e0a76347e5bac9e5d13b2da02c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f505f286631a443e9dfb99f883bfd9bb]), partition=
I20240502 08:18:18.175182 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d88c3e0a76347e5bac9e5d13b2da02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.178536 16829 tablet_bootstrap.cc:492] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:18.181665 16829 tablet_bootstrap.cc:654] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.183508 16829 tablet_bootstrap.cc:492] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:18.183677 16829 ts_tablet_manager.cc:1367] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 08:18:18.184330 16829 raft_consensus.cc:348] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.184516 16829 raft_consensus.cc:374] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.184592 16829 raft_consensus.cc:729] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.184794 16829 consensus_queue.cc:260] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.185325 16829 ts_tablet_manager.cc:1398] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:18.184975 16828 tablet_bootstrap.cc:492] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:18.187024 16828 tablet_bootstrap.cc:654] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.189585 16831 tablet_bootstrap.cc:492] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:18.191501 16831 tablet_bootstrap.cc:654] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.196745 16831 tablet_bootstrap.cc:492] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:18.196910 16831 ts_tablet_manager.cc:1367] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:18:18.197355 16828 tablet_bootstrap.cc:492] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:18.197499 16828 ts_tablet_manager.cc:1367] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20240502 08:18:18.197502 16831 raft_consensus.cc:348] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.197683 16831 raft_consensus.cc:374] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.197775 16831 raft_consensus.cc:729] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.198051 16831 consensus_queue.cc:260] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.198154 16828 raft_consensus.cc:348] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.198428 16828 raft_consensus.cc:374] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.198583 16828 raft_consensus.cc:729] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.198621 16831 ts_tablet_manager.cc:1398] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:18.198815 16828 consensus_queue.cc:260] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.199312 16828 ts_tablet_manager.cc:1398] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:18.208974 16619 consensus_queue.cc:1035] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.213698 16619 consensus_queue.cc:1035] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.316695 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:18.325472 12972 tablet_service.cc:1467] Processing CreateTablet for tablet e61dddb6b7084e1f8949e1cd38a2997c (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b6ad25e45eb542c3ab28cb99d251a5d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.325619 15865 tablet_service.cc:1467] Processing CreateTablet for tablet e61dddb6b7084e1f8949e1cd38a2997c (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b6ad25e45eb542c3ab28cb99d251a5d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.325908 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61dddb6b7084e1f8949e1cd38a2997c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.326004 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61dddb6b7084e1f8949e1cd38a2997c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.326354 12709 tablet_service.cc:1467] Processing CreateTablet for tablet e61dddb6b7084e1f8949e1cd38a2997c (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b6ad25e45eb542c3ab28cb99d251a5d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.326942 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61dddb6b7084e1f8949e1cd38a2997c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.331900 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 156838b04be24853897a4fabb5916029 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b6ad25e45eb542c3ab28cb99d251a5d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.332314 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156838b04be24853897a4fabb5916029. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.332288 15867 tablet_service.cc:1467] Processing CreateTablet for tablet 156838b04be24853897a4fabb5916029 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b6ad25e45eb542c3ab28cb99d251a5d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.332374 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 156838b04be24853897a4fabb5916029 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b6ad25e45eb542c3ab28cb99d251a5d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.332659 15867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156838b04be24853897a4fabb5916029. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.332727 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156838b04be24853897a4fabb5916029. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.335611 15468 tablet_service.cc:1467] Processing CreateTablet for tablet a9f90ed1e4e04c93995a5abba79ecb34 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b6ad25e45eb542c3ab28cb99d251a5d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.335991 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f90ed1e4e04c93995a5abba79ecb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.332257 15866 tablet_service.cc:1467] Processing CreateTablet for tablet a9f90ed1e4e04c93995a5abba79ecb34 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b6ad25e45eb542c3ab28cb99d251a5d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.336704 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f90ed1e4e04c93995a5abba79ecb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.337785 16839 tablet_bootstrap.cc:492] T e61dddb6b7084e1f8949e1cd38a2997c P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:18.338268 15601 tablet_service.cc:1467] Processing CreateTablet for tablet a9f90ed1e4e04c93995a5abba79ecb34 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=b6ad25e45eb542c3ab28cb99d251a5d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.338618 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f90ed1e4e04c93995a5abba79ecb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.339074 16829 tablet_bootstrap.cc:492] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:18.339721 16839 tablet_bootstrap.cc:654] T e61dddb6b7084e1f8949e1cd38a2997c P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.340463 16829 tablet_bootstrap.cc:654] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.347852 16831 tablet_bootstrap.cc:492] T a9f90ed1e4e04c93995a5abba79ecb34 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:18.349269 16831 tablet_bootstrap.cc:654] T a9f90ed1e4e04c93995a5abba79ecb34 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.349742 16828 tablet_bootstrap.cc:492] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:18.350008 16829 tablet_bootstrap.cc:492] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:18.350163 16829 ts_tablet_manager.cc:1367] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240502 08:18:18.350908 16829 raft_consensus.cc:348] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.351258 16828 tablet_bootstrap.cc:654] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.351133 16829 raft_consensus.cc:374] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.352465 16829 raft_consensus.cc:729] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.352710 16829 consensus_queue.cc:260] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.353230 16829 ts_tablet_manager.cc:1398] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:18.361271 16842 tablet_bootstrap.cc:492] T a9f90ed1e4e04c93995a5abba79ecb34 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:18.362341 16828 tablet_bootstrap.cc:492] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:18.362571 16828 ts_tablet_manager.cc:1367] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 08:18:18.363102 16842 tablet_bootstrap.cc:654] T a9f90ed1e4e04c93995a5abba79ecb34 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.363391 16828 raft_consensus.cc:348] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:18.363662 16828 raft_consensus.cc:374] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.363749 16828 raft_consensus.cc:729] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.363970 16828 consensus_queue.cc:260] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:18.364504 16828 ts_tablet_manager.cc:1398] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:18.364797 16828 tablet_bootstrap.cc:492] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:18.366178 16828 tablet_bootstrap.cc:654] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.369225 16831 tablet_bootstrap.cc:492] T a9f90ed1e4e04c93995a5abba79ecb34 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:18.369442 16831 ts_tablet_manager.cc:1367] T a9f90ed1e4e04c93995a5abba79ecb34 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.004s
I20240502 08:18:18.369714 16845 tablet_bootstrap.cc:492] T e61dddb6b7084e1f8949e1cd38a2997c P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:18.370231 16828 tablet_bootstrap.cc:492] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:18.370370 16828 ts_tablet_manager.cc:1367] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:18:18.370375 16831 raft_consensus.cc:348] T a9f90ed1e4e04c93995a5abba79ecb34 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:18.370697 16831 raft_consensus.cc:374] T a9f90ed1e4e04c93995a5abba79ecb34 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.370826 16831 raft_consensus.cc:729] T a9f90ed1e4e04c93995a5abba79ecb34 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.371170 16831 consensus_queue.cc:260] T a9f90ed1e4e04c93995a5abba79ecb34 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:18.371498 16845 tablet_bootstrap.cc:654] T e61dddb6b7084e1f8949e1cd38a2997c P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.374001 16828 raft_consensus.cc:348] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.374187 16828 raft_consensus.cc:374] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.374276 16828 raft_consensus.cc:729] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.374516 16828 consensus_queue.cc:260] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.375114 16828 ts_tablet_manager.cc:1398] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:18.375387 16828 tablet_bootstrap.cc:492] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:18.376920 16828 tablet_bootstrap.cc:654] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.378777 16839 tablet_bootstrap.cc:492] T e61dddb6b7084e1f8949e1cd38a2997c P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:18.378978 16839 ts_tablet_manager.cc:1367] T e61dddb6b7084e1f8949e1cd38a2997c P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.018s
I20240502 08:18:18.379940 16839 raft_consensus.cc:348] T e61dddb6b7084e1f8949e1cd38a2997c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:18.380208 16839 raft_consensus.cc:374] T e61dddb6b7084e1f8949e1cd38a2997c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.380326 16839 raft_consensus.cc:729] T e61dddb6b7084e1f8949e1cd38a2997c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.380570 16839 consensus_queue.cc:260] T e61dddb6b7084e1f8949e1cd38a2997c P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:18.381147 16831 ts_tablet_manager.cc:1398] T a9f90ed1e4e04c93995a5abba79ecb34 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 08:18:18.382130 16839 ts_tablet_manager.cc:1398] T e61dddb6b7084e1f8949e1cd38a2997c P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:18.383124 16845 tablet_bootstrap.cc:492] T e61dddb6b7084e1f8949e1cd38a2997c P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:18.383276 16845 ts_tablet_manager.cc:1367] T e61dddb6b7084e1f8949e1cd38a2997c P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 08:18:18.383576 16831 tablet_bootstrap.cc:492] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:18.385143 16831 tablet_bootstrap.cc:654] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.385861 16845 raft_consensus.cc:348] T e61dddb6b7084e1f8949e1cd38a2997c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:18.386127 16845 raft_consensus.cc:374] T e61dddb6b7084e1f8949e1cd38a2997c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.386270 16845 raft_consensus.cc:729] T e61dddb6b7084e1f8949e1cd38a2997c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.386592 16845 consensus_queue.cc:260] T e61dddb6b7084e1f8949e1cd38a2997c P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:18.388381 16845 ts_tablet_manager.cc:1398] T e61dddb6b7084e1f8949e1cd38a2997c P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:18.391783 16831 tablet_bootstrap.cc:492] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:18.391935 16831 ts_tablet_manager.cc:1367] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:18:18.392661 16831 raft_consensus.cc:348] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.392989 16831 raft_consensus.cc:374] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.393111 16831 raft_consensus.cc:729] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.393390 16831 consensus_queue.cc:260] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.394982 16831 ts_tablet_manager.cc:1398] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:18.396375 16828 tablet_bootstrap.cc:492] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:18.397063 16828 ts_tablet_manager.cc:1367] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.002s
I20240502 08:18:18.397612 16842 tablet_bootstrap.cc:492] T a9f90ed1e4e04c93995a5abba79ecb34 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:18.397825 16842 ts_tablet_manager.cc:1367] T a9f90ed1e4e04c93995a5abba79ecb34 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.012s
I20240502 08:18:18.397907 16828 raft_consensus.cc:348] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:18.398144 16828 raft_consensus.cc:374] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.398249 16828 raft_consensus.cc:729] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.398556 16828 consensus_queue.cc:260] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:18.399324 16828 ts_tablet_manager.cc:1398] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:18.399549 16842 raft_consensus.cc:348] T a9f90ed1e4e04c93995a5abba79ecb34 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:18.399884 16842 raft_consensus.cc:374] T a9f90ed1e4e04c93995a5abba79ecb34 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.400193 16842 raft_consensus.cc:729] T a9f90ed1e4e04c93995a5abba79ecb34 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.400516 16842 consensus_queue.cc:260] T a9f90ed1e4e04c93995a5abba79ecb34 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:18.401183 16842 ts_tablet_manager.cc:1398] T a9f90ed1e4e04c93995a5abba79ecb34 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:18.458549 16632 raft_consensus.cc:482] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:18.458720 16632 raft_consensus.cc:504] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:18.459353 16632 leader_election.cc:288] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:18.459704 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f90ed1e4e04c93995a5abba79ecb34" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:18.459869 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f90ed1e4e04c93995a5abba79ecb34" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:18.459935 15621 raft_consensus.cc:2454] T a9f90ed1e4e04c93995a5abba79ecb34 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:18.460110 15487 raft_consensus.cc:2454] T a9f90ed1e4e04c93995a5abba79ecb34 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:18.460402 15822 leader_election.cc:302] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:18.460645 16632 raft_consensus.cc:2781] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:18.460736 16632 raft_consensus.cc:482] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:18.460816 16632 raft_consensus.cc:3037] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.462824 16632 raft_consensus.cc:504] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:18.463419 16632 leader_election.cc:288] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:18.463869 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f90ed1e4e04c93995a5abba79ecb34" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:18.463899 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f90ed1e4e04c93995a5abba79ecb34" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:18.464025 15621 raft_consensus.cc:3037] T a9f90ed1e4e04c93995a5abba79ecb34 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.464090 15487 raft_consensus.cc:3037] T a9f90ed1e4e04c93995a5abba79ecb34 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.465968 15621 raft_consensus.cc:2454] T a9f90ed1e4e04c93995a5abba79ecb34 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:18.466090 15487 raft_consensus.cc:2454] T a9f90ed1e4e04c93995a5abba79ecb34 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:18.466416 15822 leader_election.cc:302] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:18.466869 16632 raft_consensus.cc:2781] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:18.467001 16632 raft_consensus.cc:686] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:18.467244 16632 consensus_queue.cc:237] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:18.470155 12584 catalog_manager.cc:5484] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:18.496766 16580 raft_consensus.cc:482] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:18.496943 16580 raft_consensus.cc:504] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.497563 16580 leader_election.cc:288] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:18.497929 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d88c3e0a76347e5bac9e5d13b2da02c" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:18.498019 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d88c3e0a76347e5bac9e5d13b2da02c" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:18.498178 15754 raft_consensus.cc:2454] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:18.498262 15887 raft_consensus.cc:2454] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:18.498667 15553 leader_election.cc:302] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:18.498945 16580 raft_consensus.cc:2781] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:18.499058 16580 raft_consensus.cc:482] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:18.499147 16580 raft_consensus.cc:3037] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.501178 16580 raft_consensus.cc:504] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.501668 16580 leader_election.cc:288] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:18.502056 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d88c3e0a76347e5bac9e5d13b2da02c" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:18.502121 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d88c3e0a76347e5bac9e5d13b2da02c" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:18.502264 15754 raft_consensus.cc:3037] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.502290 15887 raft_consensus.cc:3037] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.504961 15887 raft_consensus.cc:2454] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:18.504949 15754 raft_consensus.cc:2454] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:18.505342 15553 leader_election.cc:302] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:18.505872 16580 raft_consensus.cc:2781] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:18.506114 16580 raft_consensus.cc:686] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:18.506544 16580 consensus_queue.cc:237] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.509505 12584 catalog_manager.cc:5484] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:18.613554 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:18.619571 16816 raft_consensus.cc:482] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:18.619731 16816 raft_consensus.cc:504] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.619840 15886 raft_consensus.cc:1261] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:18.620461 16816 leader_election.cc:288] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:18.620720 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156838b04be24853897a4fabb5916029" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:18.620801 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156838b04be24853897a4fabb5916029" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:18.620924 15886 raft_consensus.cc:2454] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:18.621012 15621 raft_consensus.cc:2454] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:18.621290 15689 leader_election.cc:302] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:18.621532 16816 raft_consensus.cc:2781] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:18.621639 16816 raft_consensus.cc:482] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:18.621718 16816 raft_consensus.cc:3037] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.624459 16816 raft_consensus.cc:504] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.625051 16816 leader_election.cc:288] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:18.625267 15754 raft_consensus.cc:1261] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:18.625527 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156838b04be24853897a4fabb5916029" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:18.625689 15886 raft_consensus.cc:3037] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.626446 16580 consensus_queue.cc:1035] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.627736 15886 raft_consensus.cc:2454] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:18.631326 12579 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 7d88c3e0a76347e5bac9e5d13b2da02c (table rebalancer_dynamic_table [id=f505f286631a443e9dfb99f883bfd9bb]). Expected version 1 got 0
I20240502 08:18:18.632838 16580 consensus_queue.cc:1035] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.632946 16857 tablet.cc:1722] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:18.633565 15689 leader_election.cc:302] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:18.633625 16857 tablet.cc:1620] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:18.634232 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156838b04be24853897a4fabb5916029" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:18.634090 16854 raft_consensus.cc:2781] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:18.634454 15621 raft_consensus.cc:3037] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.634483 16854 raft_consensus.cc:686] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:18.634796 16854 consensus_queue.cc:237] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:18.637444 15621 raft_consensus.cc:2454] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:18.639185 12579 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 7d88c3e0a76347e5bac9e5d13b2da02c (table rebalancer_dynamic_table [id=f505f286631a443e9dfb99f883bfd9bb]). Expected version 1 got 0
I20240502 08:18:18.644994 16856 tablet.cc:1722] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:18.645527 16856 tablet.cc:1620] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:18.646003 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=f505f286631a443e9dfb99f883bfd9bb] alter complete (version 1)
I20240502 08:18:18.654546 16862 tablet.cc:1722] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:18.655052 16862 tablet.cc:1620] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:18.657553 12584 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet 7d88c3e0a76347e5bac9e5d13b2da02c (table rebalancer_dynamic_table [id=f505f286631a443e9dfb99f883bfd9bb]). Expected version 1 got 0
I20240502 08:18:18.661033 12584 catalog_manager.cc:5484] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:18.689241 16767 raft_consensus.cc:482] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:18.689404 16767 raft_consensus.cc:504] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:18.690055 16767 leader_election.cc:288] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:18.690366 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61dddb6b7084e1f8949e1cd38a2997c" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:18.690634 12992 raft_consensus.cc:2454] T e61dddb6b7084e1f8949e1cd38a2997c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:18.690685 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61dddb6b7084e1f8949e1cd38a2997c" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:18.690907 12727 raft_consensus.cc:2454] T e61dddb6b7084e1f8949e1cd38a2997c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:18.691041 15819 leader_election.cc:302] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:18:18.691331 16767 raft_consensus.cc:2781] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:18.691449 16767 raft_consensus.cc:482] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:18.691537 16767 raft_consensus.cc:3037] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.693527 16767 raft_consensus.cc:504] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:18.693964 16767 leader_election.cc:288] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:18.694317 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61dddb6b7084e1f8949e1cd38a2997c" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:18.694427 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61dddb6b7084e1f8949e1cd38a2997c" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:18.694518 12992 raft_consensus.cc:3037] T e61dddb6b7084e1f8949e1cd38a2997c P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.694656 12727 raft_consensus.cc:3037] T e61dddb6b7084e1f8949e1cd38a2997c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.697175 12992 raft_consensus.cc:2454] T e61dddb6b7084e1f8949e1cd38a2997c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:18.697189 12727 raft_consensus.cc:2454] T e61dddb6b7084e1f8949e1cd38a2997c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:18.697526 15819 leader_election.cc:302] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:18:18.697762 16767 raft_consensus.cc:2781] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:18.697896 16767 raft_consensus.cc:686] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:18.698138 16767 consensus_queue.cc:237] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:18.701121 12584 catalog_manager.cc:5484] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:18.748003 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:18.754783 16856 tablet.cc:1722] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:18.755259 16856 tablet.cc:1620] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:18.756810 16862 tablet.cc:1722] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:18.757237 16862 tablet.cc:1620] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:18.757997 16857 tablet.cc:1722] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:18.758419 16857 tablet.cc:1620] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:18.764755 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=f505f286631a443e9dfb99f883bfd9bb] alter complete (version 2)
I20240502 08:18:18.867087 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:18.873854 16856 tablet.cc:1722] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:18.874608 16857 tablet.cc:1722] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:18.875169 16862 tablet.cc:1722] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:18.884162 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=f505f286631a443e9dfb99f883bfd9bb] alter complete (version 3)
I20240502 08:18:18.894527 16767 consensus_queue.cc:1035] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.896710 16767 consensus_queue.cc:1035] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.986984 12586 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:18.987183 12586 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:18.990202 12586 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 7d88c3e0a76347e5bac9e5d13b2da02c
I20240502 08:18:18.990936 15734 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d88c3e0a76347e5bac9e5d13b2da02c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:18 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:18.991076 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d88c3e0a76347e5bac9e5d13b2da02c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:18 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:18.991361 15601 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d88c3e0a76347e5bac9e5d13b2da02c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:18 UTC) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:18.991683 16868 tablet_replica.cc:330] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:18.991806 16869 tablet_replica.cc:330] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:18.992049 16869 raft_consensus.cc:2229] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:18.992076 16868 raft_consensus.cc:2229] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:18.992306 16869 raft_consensus.cc:2258] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:18.992484 16868 raft_consensus.cc:2258] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:18.993242 16869 ts_tablet_manager.cc:1875] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:18.993295 16868 ts_tablet_manager.cc:1875] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:18.993633 16870 tablet_replica.cc:330] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:18.993821 16870 raft_consensus.cc:2229] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:18.994112 16870 raft_consensus.cc:2258] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:18.994747 16870 ts_tablet_manager.cc:1875] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:18.998844 16870 ts_tablet_manager.cc:1888] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:18.998994 16870 log.cc:1196] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/7d88c3e0a76347e5bac9e5d13b2da02c
I20240502 08:18:18.999264 16869 ts_tablet_manager.cc:1888] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:18.999420 16869 log.cc:1196] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/7d88c3e0a76347e5bac9e5d13b2da02c
I20240502 08:18:18.999490 16870 ts_tablet_manager.cc:1909] T 7d88c3e0a76347e5bac9e5d13b2da02c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting consensus metadata
I20240502 08:18:18.999840 16869 ts_tablet_manager.cc:1909] T 7d88c3e0a76347e5bac9e5d13b2da02c P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:19.000034 16868 ts_tablet_manager.cc:1888] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:19.000172 16868 log.cc:1196] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/7d88c3e0a76347e5bac9e5d13b2da02c
I20240502 08:18:19.000617 16868 ts_tablet_manager.cc:1909] T 7d88c3e0a76347e5bac9e5d13b2da02c P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:19.000833 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 7d88c3e0a76347e5bac9e5d13b2da02c (table rebalancer_dynamic_table_new_name [id=f505f286631a443e9dfb99f883bfd9bb]) successfully deleted
I20240502 08:18:19.001118 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 7d88c3e0a76347e5bac9e5d13b2da02c (table rebalancer_dynamic_table_new_name [id=f505f286631a443e9dfb99f883bfd9bb]) successfully deleted
I20240502 08:18:19.001677 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 7d88c3e0a76347e5bac9e5d13b2da02c (table rebalancer_dynamic_table_new_name [id=f505f286631a443e9dfb99f883bfd9bb]) successfully deleted
I20240502 08:18:19.067564 16816 consensus_queue.cc:1035] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.071619 16816 consensus_queue.cc:1035] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.092900 12586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:19.099615 15865 tablet_service.cc:1467] Processing CreateTablet for tablet ddaf1fdeb2cb4f04a0850956c68a8521 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=df86e2d54682489da9ac79abe63332df]), partition=
I20240502 08:18:19.099953 15601 tablet_service.cc:1467] Processing CreateTablet for tablet ddaf1fdeb2cb4f04a0850956c68a8521 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=df86e2d54682489da9ac79abe63332df]), partition=
I20240502 08:18:19.100029 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddaf1fdeb2cb4f04a0850956c68a8521. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.100296 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddaf1fdeb2cb4f04a0850956c68a8521. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.104625 16873 tablet_bootstrap.cc:492] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:19.106086 16873 tablet_bootstrap.cc:654] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.106329 16874 tablet_bootstrap.cc:492] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:19.106370 13102 tablet_service.cc:1467] Processing CreateTablet for tablet ddaf1fdeb2cb4f04a0850956c68a8521 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=df86e2d54682489da9ac79abe63332df]), partition=
I20240502 08:18:19.106734 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddaf1fdeb2cb4f04a0850956c68a8521. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.107836 16874 tablet_bootstrap.cc:654] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.109884 16874 tablet_bootstrap.cc:492] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:19.110010 16874 ts_tablet_manager.cc:1367] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:19.110734 16874 raft_consensus.cc:348] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:19.110939 16874 raft_consensus.cc:374] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.111032 16874 raft_consensus.cc:729] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.111248 16874 consensus_queue.cc:260] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:19.112766 16874 ts_tablet_manager.cc:1398] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:19.115900 16878 tablet_bootstrap.cc:492] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:19.116309 16873 tablet_bootstrap.cc:492] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:19.116470 16873 ts_tablet_manager.cc:1367] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240502 08:18:19.117035 16873 raft_consensus.cc:348] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:19.117205 16873 raft_consensus.cc:374] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.117220 16878 tablet_bootstrap.cc:654] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.117293 16873 raft_consensus.cc:729] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.117497 16873 consensus_queue.cc:260] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:19.118000 16873 ts_tablet_manager.cc:1398] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:18:19.118939 16878 tablet_bootstrap.cc:492] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:19.119097 16878 ts_tablet_manager.cc:1367] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 08:18:19.119731 16878 raft_consensus.cc:348] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:19.119880 16878 raft_consensus.cc:374] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.119992 16878 raft_consensus.cc:729] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.120187 16878 consensus_queue.cc:260] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:19.120716 16878 ts_tablet_manager.cc:1398] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:19.148471 16865 consensus_queue.cc:1035] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.152204 16767 consensus_queue.cc:1035] T e61dddb6b7084e1f8949e1cd38a2997c P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.161897 16767 raft_consensus.cc:482] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:19.162036 16767 raft_consensus.cc:504] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:19.162690 16767 leader_election.cc:288] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:19.163216 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddaf1fdeb2cb4f04a0850956c68a8521" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:19.163430 15621 raft_consensus.cc:2454] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:19.163902 15822 leader_election.cc:302] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:19.164163 16865 raft_consensus.cc:2781] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:19.164278 16865 raft_consensus.cc:482] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:19.164367 16865 raft_consensus.cc:3037] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.166580 16865 raft_consensus.cc:504] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:19.167220 16865 leader_election.cc:288] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:19.167724 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddaf1fdeb2cb4f04a0850956c68a8521" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:19.167922 15621 raft_consensus.cc:3037] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.170035 15621 raft_consensus.cc:2454] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:19.170209 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddaf1fdeb2cb4f04a0850956c68a8521" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:19.170397 13119 raft_consensus.cc:3037] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.172173 13119 raft_consensus.cc:2454] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:19.172461 15822 leader_election.cc:302] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:19.170166 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddaf1fdeb2cb4f04a0850956c68a8521" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:19.172735 16865 raft_consensus.cc:2781] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:19.172869 16865 raft_consensus.cc:686] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:19.172871 13121 raft_consensus.cc:2362] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1. Re-sending same reply.
I20240502 08:18:19.173169 16865 consensus_queue.cc:237] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:19.175896 12587 catalog_manager.cc:5484] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:19.217705 12587 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:19.226672 15734 tablet_service.cc:1467] Processing CreateTablet for tablet aadd5844b106485e95ecf6fa923cd6ea (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=356db21cc0814b908e5333ba8c2e04ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.227166 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadd5844b106485e95ecf6fa923cd6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.227429 15735 tablet_service.cc:1467] Processing CreateTablet for tablet bb73ec8c37464c8ba97ac2372667525a (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=356db21cc0814b908e5333ba8c2e04ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.227489 15601 tablet_service.cc:1467] Processing CreateTablet for tablet aadd5844b106485e95ecf6fa923cd6ea (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=356db21cc0814b908e5333ba8c2e04ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.227891 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadd5844b106485e95ecf6fa923cd6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.227896 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb73ec8c37464c8ba97ac2372667525a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.231168 15468 tablet_service.cc:1467] Processing CreateTablet for tablet aadd5844b106485e95ecf6fa923cd6ea (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=356db21cc0814b908e5333ba8c2e04ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.231572 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadd5844b106485e95ecf6fa923cd6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.232167 12842 tablet_service.cc:1467] Processing CreateTablet for tablet bb73ec8c37464c8ba97ac2372667525a (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=356db21cc0814b908e5333ba8c2e04ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.232558 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb73ec8c37464c8ba97ac2372667525a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.235747 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae7246870f14d7da744a1856430487c (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=356db21cc0814b908e5333ba8c2e04ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.236155 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae7246870f14d7da744a1856430487c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.236734 15599 tablet_service.cc:1467] Processing CreateTablet for tablet bb73ec8c37464c8ba97ac2372667525a (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=356db21cc0814b908e5333ba8c2e04ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.237099 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb73ec8c37464c8ba97ac2372667525a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.238667 16887 tablet_bootstrap.cc:492] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:19.239619 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae7246870f14d7da744a1856430487c (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=356db21cc0814b908e5333ba8c2e04ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.239691 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae7246870f14d7da744a1856430487c (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=356db21cc0814b908e5333ba8c2e04ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:19.239995 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae7246870f14d7da744a1856430487c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.240015 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae7246870f14d7da744a1856430487c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.240800 16887 tablet_bootstrap.cc:654] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.251145 16873 tablet_bootstrap.cc:492] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:19.252646 16873 tablet_bootstrap.cc:654] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.254657 16874 tablet_bootstrap.cc:492] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:19.256206 16874 tablet_bootstrap.cc:654] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.260726 16891 tablet_bootstrap.cc:492] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:19.264762 16890 tablet_bootstrap.cc:492] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:19.266291 16891 tablet_bootstrap.cc:654] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.267427 16893 tablet_bootstrap.cc:492] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:19.268456 16890 tablet_bootstrap.cc:654] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.269047 16893 tablet_bootstrap.cc:654] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.269605 16873 tablet_bootstrap.cc:492] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:19.269822 16873 ts_tablet_manager.cc:1367] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.006s
I20240502 08:18:19.269897 16874 tablet_bootstrap.cc:492] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:19.270049 16874 ts_tablet_manager.cc:1367] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.006s
I20240502 08:18:19.270747 16873 raft_consensus.cc:348] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:19.270838 16874 raft_consensus.cc:348] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.271019 16874 raft_consensus.cc:374] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.271088 16874 raft_consensus.cc:729] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.271242 16873 raft_consensus.cc:374] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.271369 16873 raft_consensus.cc:729] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.271315 16874 consensus_queue.cc:260] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.271705 16873 consensus_queue.cc:260] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:19.272715 16874 ts_tablet_manager.cc:1398] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:19.273027 16874 tablet_bootstrap.cc:492] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:19.274715 16874 tablet_bootstrap.cc:654] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.277856 16893 tablet_bootstrap.cc:492] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:19.278003 16893 ts_tablet_manager.cc:1367] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 08:18:19.279218 16893 raft_consensus.cc:348] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:19.279415 16893 raft_consensus.cc:374] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.280802 16891 tablet_bootstrap.cc:492] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:19.280946 16891 ts_tablet_manager.cc:1367] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240502 08:18:19.281034 16873 ts_tablet_manager.cc:1398] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240502 08:18:19.281662 16891 raft_consensus.cc:348] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:19.281857 16891 raft_consensus.cc:374] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.281953 16891 raft_consensus.cc:729] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.282172 16891 consensus_queue.cc:260] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:19.283556 16891 ts_tablet_manager.cc:1398] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:19.283743 16874 tablet_bootstrap.cc:492] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:19.283771 16893 raft_consensus.cc:729] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.283919 16874 ts_tablet_manager.cc:1367] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240502 08:18:19.284076 16893 consensus_queue.cc:260] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:19.284833 16893 ts_tablet_manager.cc:1398] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:19.285033 16874 raft_consensus.cc:348] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:19.285226 16874 raft_consensus.cc:374] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.285310 16874 raft_consensus.cc:729] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.285481 16874 consensus_queue.cc:260] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:19.285987 16874 ts_tablet_manager.cc:1398] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:19.287118 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:19.292755 16887 tablet_bootstrap.cc:492] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:19.293426 16887 ts_tablet_manager.cc:1367] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.013s
I20240502 08:18:19.293689 15621 raft_consensus.cc:1261] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:19.293879 13119 raft_consensus.cc:1261] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:19.294272 16887 raft_consensus.cc:348] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.294411 16865 consensus_queue.cc:1035] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.294529 16887 raft_consensus.cc:374] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.294623 16887 raft_consensus.cc:729] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.294628 16632 consensus_queue.cc:1035] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.294860 16887 consensus_queue.cc:260] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.299981 16890 tablet_bootstrap.cc:492] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:19.300170 16890 ts_tablet_manager.cc:1367] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.000s
I20240502 08:18:19.300864 16890 raft_consensus.cc:348] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:19.301029 16890 raft_consensus.cc:374] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.301149 16890 raft_consensus.cc:729] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.301609 16890 consensus_queue.cc:260] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:19.302246 16890 ts_tablet_manager.cc:1398] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:19.301574 12579 catalog_manager.cc:5576] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983) does not have the latest schema for tablet ddaf1fdeb2cb4f04a0850956c68a8521 (table rebalancer_dynamic_table [id=df86e2d54682489da9ac79abe63332df]). Expected version 1 got 0
I20240502 08:18:19.302668 16890 tablet_bootstrap.cc:492] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:19.303556 16887 ts_tablet_manager.cc:1398] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:18:19.304221 16890 tablet_bootstrap.cc:654] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.308712 16857 tablet.cc:1722] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:19.309221 16857 tablet.cc:1620] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:19.310564 16887 tablet_bootstrap.cc:492] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:19.319463 12583 catalog_manager.cc:5576] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979) does not have the latest schema for tablet ddaf1fdeb2cb4f04a0850956c68a8521 (table rebalancer_dynamic_table [id=df86e2d54682489da9ac79abe63332df]). Expected version 1 got 0
I20240502 08:18:19.320015 16904 tablet.cc:1722] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:19.320998 16904 tablet.cc:1620] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:19.325275 16856 tablet.cc:1722] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:19.325736 16856 tablet.cc:1620] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:19.328685 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=df86e2d54682489da9ac79abe63332df] alter complete (version 1)
I20240502 08:18:19.327446 16887 tablet_bootstrap.cc:654] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.339282 16887 tablet_bootstrap.cc:492] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:19.339493 16887 ts_tablet_manager.cc:1367] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240502 08:18:19.340247 16887 raft_consensus.cc:348] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:19.340893 16887 raft_consensus.cc:374] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.341081 16887 raft_consensus.cc:729] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.341396 16887 consensus_queue.cc:260] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:19.342404 16887 ts_tablet_manager.cc:1398] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:19.347642 16890 tablet_bootstrap.cc:492] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:19.347846 16890 ts_tablet_manager.cc:1367] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.011s
I20240502 08:18:19.348588 16890 raft_consensus.cc:348] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.348855 16890 raft_consensus.cc:374] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.348958 16890 raft_consensus.cc:729] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.349251 16890 consensus_queue.cc:260] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.349813 16890 ts_tablet_manager.cc:1398] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:19.434521 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:19.442723 16857 tablet.cc:1722] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:19.443243 16857 tablet.cc:1620] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:19.443643 16856 tablet.cc:1722] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:19.444062 16856 tablet.cc:1620] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:19.444000 16904 tablet.cc:1722] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:19.444456 16904 tablet.cc:1620] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:19.453886 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=df86e2d54682489da9ac79abe63332df] alter complete (version 2)
I20240502 08:18:19.466279 16759 raft_consensus.cc:482] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:19.466485 16759 raft_consensus.cc:504] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:19.467245 16759 leader_election.cc:288] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:19.467593 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae7246870f14d7da744a1856430487c" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:19.467836 15487 raft_consensus.cc:2454] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:19.467866 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae7246870f14d7da744a1856430487c" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:19.468111 15887 raft_consensus.cc:2454] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:19.468216 12664 leader_election.cc:302] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:19.468494 16759 raft_consensus.cc:2781] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:19.468603 16759 raft_consensus.cc:482] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:19.468688 16759 raft_consensus.cc:3037] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.470863 16759 raft_consensus.cc:504] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:19.471482 16759 leader_election.cc:288] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:19.471958 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae7246870f14d7da744a1856430487c" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:19.471952 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae7246870f14d7da744a1856430487c" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:19.472182 15887 raft_consensus.cc:3037] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.472182 15487 raft_consensus.cc:3037] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.474949 15887 raft_consensus.cc:2454] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:19.474949 15487 raft_consensus.cc:2454] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:19.475356 12664 leader_election.cc:302] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:19.475710 16759 raft_consensus.cc:2781] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:19.475889 16759 raft_consensus.cc:686] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:18:19.476214 16759 consensus_queue.cc:237] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:19.479835 12579 catalog_manager.cc:5484] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:19.525749 16854 raft_consensus.cc:482] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:19.525915 16854 raft_consensus.cc:504] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.526643 16854 leader_election.cc:288] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:19.527002 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadd5844b106485e95ecf6fa923cd6ea" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:19.527221 15621 raft_consensus.cc:2454] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:19.527200 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadd5844b106485e95ecf6fa923cd6ea" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:19.527405 15487 raft_consensus.cc:2454] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:19.527611 15690 leader_election.cc:302] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:19.527988 16854 raft_consensus.cc:2781] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:19.528065 16854 raft_consensus.cc:482] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:19.528136 16854 raft_consensus.cc:3037] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.530182 16854 raft_consensus.cc:504] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.530702 16854 leader_election.cc:288] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:19.531155 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadd5844b106485e95ecf6fa923cd6ea" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:19.531227 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadd5844b106485e95ecf6fa923cd6ea" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:19.531340 15621 raft_consensus.cc:3037] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.531392 15487 raft_consensus.cc:3037] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.533933 15487 raft_consensus.cc:2454] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:19.533922 15621 raft_consensus.cc:2454] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:19.534261 15688 leader_election.cc:302] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:19.534538 16854 raft_consensus.cc:2781] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:19.534678 16854 raft_consensus.cc:686] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:19.534911 16854 consensus_queue.cc:237] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.537919 12579 catalog_manager.cc:5484] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:19.556200 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:19.560674 16580 raft_consensus.cc:482] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:19.560880 16580 raft_consensus.cc:504] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:19.561626 16580 leader_election.cc:288] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:19.562135 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb73ec8c37464c8ba97ac2372667525a" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:19.562350 15754 raft_consensus.cc:2454] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:19.562333 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb73ec8c37464c8ba97ac2372667525a" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:19.562554 12859 raft_consensus.cc:2454] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:19.562953 15553 leader_election.cc:302] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:19.563190 16857 tablet.cc:1722] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:19.563442 16580 raft_consensus.cc:2781] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:19.563545 16580 raft_consensus.cc:482] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:19.563661 16580 raft_consensus.cc:3037] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.564072 16856 tablet.cc:1722] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:19.565171 16904 tablet.cc:1722] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:19.566207 16580 raft_consensus.cc:504] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:19.567047 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb73ec8c37464c8ba97ac2372667525a" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:19.567234 15754 raft_consensus.cc:3037] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.569753 15754 raft_consensus.cc:2454] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:19.570124 15553 leader_election.cc:302] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:19.571018 16908 raft_consensus.cc:2781] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:19.571377 16580 leader_election.cc:288] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:19.571425 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb73ec8c37464c8ba97ac2372667525a" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:19.571586 12859 raft_consensus.cc:3037] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.572046 16908 raft_consensus.cc:686] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:19.572409 16908 consensus_queue.cc:237] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:19.573709 12859 raft_consensus.cc:2454] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:19.575615 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=df86e2d54682489da9ac79abe63332df] alter complete (version 3)
I20240502 08:18:19.575688 12583 catalog_manager.cc:5484] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:19.677443 12579 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:19.677645 12579 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:19.680518 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet ddaf1fdeb2cb4f04a0850956c68a8521
I20240502 08:18:19.681219 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet ddaf1fdeb2cb4f04a0850956c68a8521 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:19 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:19.681305 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet ddaf1fdeb2cb4f04a0850956c68a8521 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:19 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:19.681614 15601 tablet_service.cc:1514] Processing DeleteTablet for tablet ddaf1fdeb2cb4f04a0850956c68a8521 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:19 UTC) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:19.681932 16912 tablet_replica.cc:330] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:19.682155 16911 tablet_replica.cc:330] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:19.682294 16912 raft_consensus.cc:2229] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:19.682381 16911 raft_consensus.cc:2229] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:19.682631 16912 raft_consensus.cc:2258] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:19.682771 16911 raft_consensus.cc:2258] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:19.683646 16912 ts_tablet_manager.cc:1875] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:19.683655 16911 ts_tablet_manager.cc:1875] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:19.690389 16912 ts_tablet_manager.cc:1888] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:19.690635 16912 log.cc:1196] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/ddaf1fdeb2cb4f04a0850956c68a8521
I20240502 08:18:19.691121 16912 ts_tablet_manager.cc:1909] T ddaf1fdeb2cb4f04a0850956c68a8521 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:19.691567 16911 ts_tablet_manager.cc:1888] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:19.691622 16913 tablet_replica.cc:330] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:19.691735 16911 log.cc:1196] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/ddaf1fdeb2cb4f04a0850956c68a8521
I20240502 08:18:19.691931 16913 raft_consensus.cc:2229] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:19.692183 16913 raft_consensus.cc:2258] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:19.692684 16911 ts_tablet_manager.cc:1909] T ddaf1fdeb2cb4f04a0850956c68a8521 P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:19.693121 16913 ts_tablet_manager.cc:1875] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:19.693748 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet ddaf1fdeb2cb4f04a0850956c68a8521 (table rebalancer_dynamic_table_new_name [id=df86e2d54682489da9ac79abe63332df]) successfully deleted
I20240502 08:18:19.695631 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet ddaf1fdeb2cb4f04a0850956c68a8521 (table rebalancer_dynamic_table_new_name [id=df86e2d54682489da9ac79abe63332df]) successfully deleted
I20240502 08:18:19.699062 16913 ts_tablet_manager.cc:1888] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:19.699175 16913 log.cc:1196] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/ddaf1fdeb2cb4f04a0850956c68a8521
I20240502 08:18:19.699525 16913 ts_tablet_manager.cc:1909] T ddaf1fdeb2cb4f04a0850956c68a8521 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting consensus metadata
I20240502 08:18:19.700294 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet ddaf1fdeb2cb4f04a0850956c68a8521 (table rebalancer_dynamic_table_new_name [id=df86e2d54682489da9ac79abe63332df]) successfully deleted
I20240502 08:18:19.782898 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:19.789764 12842 tablet_service.cc:1467] Processing CreateTablet for tablet ab14ea85b65e47c1b857a4250407980f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=121220bb856b46f39c335e87a2aee3c7]), partition=
I20240502 08:18:19.789914 15734 tablet_service.cc:1467] Processing CreateTablet for tablet ab14ea85b65e47c1b857a4250407980f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=121220bb856b46f39c335e87a2aee3c7]), partition=
I20240502 08:18:19.790138 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab14ea85b65e47c1b857a4250407980f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.790225 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab14ea85b65e47c1b857a4250407980f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.794788 15467 tablet_service.cc:1467] Processing CreateTablet for tablet ab14ea85b65e47c1b857a4250407980f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=121220bb856b46f39c335e87a2aee3c7]), partition=
I20240502 08:18:19.795056 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab14ea85b65e47c1b857a4250407980f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:19.795189 16887 tablet_bootstrap.cc:492] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:19.796579 16887 tablet_bootstrap.cc:654] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.798218 16887 tablet_bootstrap.cc:492] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:19.798326 16887 ts_tablet_manager.cc:1367] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:19.799077 16887 raft_consensus.cc:348] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.799254 16887 raft_consensus.cc:374] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.799328 16887 raft_consensus.cc:729] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.799527 16887 consensus_queue.cc:260] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.800011 16887 ts_tablet_manager.cc:1398] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:19.807782 16890 tablet_bootstrap.cc:492] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:19.809113 16890 tablet_bootstrap.cc:654] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.815093 16918 tablet_bootstrap.cc:492] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:19.815706 16890 tablet_bootstrap.cc:492] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:19.815838 16890 ts_tablet_manager.cc:1367] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240502 08:18:19.816493 16890 raft_consensus.cc:348] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.816684 16890 raft_consensus.cc:374] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.816764 16890 raft_consensus.cc:729] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.816962 16890 consensus_queue.cc:260] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.817257 16918 tablet_bootstrap.cc:654] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:19.817446 16890 ts_tablet_manager.cc:1398] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:19.818976 16918 tablet_bootstrap.cc:492] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:19.819130 16918 ts_tablet_manager.cc:1367] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:19.819641 16918 raft_consensus.cc:348] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.819795 16918 raft_consensus.cc:374] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:19.819906 16918 raft_consensus.cc:729] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:19.820145 16918 consensus_queue.cc:260] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.820684 16918 ts_tablet_manager.cc:1398] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:18:19.897945 16777 raft_consensus.cc:482] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:19.898113 16777 raft_consensus.cc:504] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.898974 16777 leader_election.cc:288] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:19.899560 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab14ea85b65e47c1b857a4250407980f" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:19.899578 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab14ea85b65e47c1b857a4250407980f" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:19.899749 12859 raft_consensus.cc:2454] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:19.899778 15754 raft_consensus.cc:2454] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:19.900137 15419 leader_election.cc:302] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db7893bb29a047b3b82cc60bfd8930a3, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:19.900540 16777 raft_consensus.cc:2781] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:19.900653 16777 raft_consensus.cc:482] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:19.900723 16777 raft_consensus.cc:3037] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.902685 16777 raft_consensus.cc:504] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.903442 16777 leader_election.cc:288] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:19.903541 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab14ea85b65e47c1b857a4250407980f" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:19.903725 12859 raft_consensus.cc:3037] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.903729 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab14ea85b65e47c1b857a4250407980f" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:19.903911 15754 raft_consensus.cc:3037] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:19.905977 12859 raft_consensus.cc:2454] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:19.906140 15754 raft_consensus.cc:2454] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:19.906551 15418 leader_election.cc:302] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:19.907291 16777 raft_consensus.cc:2781] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:19.907469 16777 raft_consensus.cc:686] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:19.907709 16777 consensus_queue.cc:237] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:19.910430 12583 catalog_manager.cc:5484] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:19.916903 16816 consensus_queue.cc:1035] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.920581 16816 consensus_queue.cc:1035] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.966181 16908 consensus_queue.cc:1035] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.967387 16809 consensus_queue.cc:1035] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.971822 16909 consensus_queue.cc:1035] T bb73ec8c37464c8ba97ac2372667525a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:19.975718 16759 consensus_queue.cc:1035] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.043912 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:20.049976 15754 raft_consensus.cc:1261] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:20.050098 12859 raft_consensus.cc:1261] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:20.050619 16915 consensus_queue.cc:1035] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.050863 16777 consensus_queue.cc:1035] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.054414 16935 tablet.cc:1722] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:20.055194 16935 tablet.cc:1620] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:20.055550 16936 tablet.cc:1722] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:20.056110 16936 tablet.cc:1620] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:20.056602 12583 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet ab14ea85b65e47c1b857a4250407980f (table rebalancer_dynamic_table [id=121220bb856b46f39c335e87a2aee3c7]). Expected version 1 got 0
I20240502 08:18:20.058743 12579 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet ab14ea85b65e47c1b857a4250407980f (table rebalancer_dynamic_table [id=121220bb856b46f39c335e87a2aee3c7]). Expected version 1 got 0
I20240502 08:18:20.061427 16937 tablet.cc:1722] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:20.062235 16937 tablet.cc:1620] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:20.064594 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=121220bb856b46f39c335e87a2aee3c7] alter complete (version 1)
I20240502 08:18:20.065508 12587 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet ab14ea85b65e47c1b857a4250407980f (table rebalancer_dynamic_table [id=121220bb856b46f39c335e87a2aee3c7]). Expected version 1 got 0
I20240502 08:18:20.125018 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:20.133774 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 438591418b6044f0abbe4d56bba217a2 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9326d144342b43d1ab9279d9cca295a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:20.133967 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 438591418b6044f0abbe4d56bba217a2 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9326d144342b43d1ab9279d9cca295a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:20.134214 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438591418b6044f0abbe4d56bba217a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.134366 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438591418b6044f0abbe4d56bba217a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.134402 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6a660632634e3c9abde21dbf84dfc0 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9326d144342b43d1ab9279d9cca295a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:20.134892 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6a660632634e3c9abde21dbf84dfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.135067 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 438591418b6044f0abbe4d56bba217a2 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9326d144342b43d1ab9279d9cca295a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:20.135497 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 438591418b6044f0abbe4d56bba217a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.138813 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 82ecdc518ec94e43b0867a0786575304 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9326d144342b43d1ab9279d9cca295a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:20.138839 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6a660632634e3c9abde21dbf84dfc0 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9326d144342b43d1ab9279d9cca295a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:20.139117 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ecdc518ec94e43b0867a0786575304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.139315 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6a660632634e3c9abde21dbf84dfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.140033 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6a660632634e3c9abde21dbf84dfc0 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9326d144342b43d1ab9279d9cca295a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:20.140410 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6a660632634e3c9abde21dbf84dfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.145877 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 82ecdc518ec94e43b0867a0786575304 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9326d144342b43d1ab9279d9cca295a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:20.147069 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 82ecdc518ec94e43b0867a0786575304 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=9326d144342b43d1ab9279d9cca295a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:20.147490 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ecdc518ec94e43b0867a0786575304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.150287 16887 tablet_bootstrap.cc:492] T 438591418b6044f0abbe4d56bba217a2 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:20.151975 16887 tablet_bootstrap.cc:654] T 438591418b6044f0abbe4d56bba217a2 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.152397 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ecdc518ec94e43b0867a0786575304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.154995 16887 tablet_bootstrap.cc:492] T 438591418b6044f0abbe4d56bba217a2 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:20.155149 16887 ts_tablet_manager.cc:1367] T 438591418b6044f0abbe4d56bba217a2 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 08:18:20.155694 16940 tablet_bootstrap.cc:492] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:20.155828 16887 raft_consensus.cc:348] T 438591418b6044f0abbe4d56bba217a2 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:20.156069 16887 raft_consensus.cc:374] T 438591418b6044f0abbe4d56bba217a2 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.156143 16887 raft_consensus.cc:729] T 438591418b6044f0abbe4d56bba217a2 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.156368 16887 consensus_queue.cc:260] T 438591418b6044f0abbe4d56bba217a2 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:20.156905 16887 ts_tablet_manager.cc:1398] T 438591418b6044f0abbe4d56bba217a2 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:20.157192 16940 tablet_bootstrap.cc:654] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.157634 16939 tablet_bootstrap.cc:492] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:20.161399 16939 tablet_bootstrap.cc:654] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.170770 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:20.174593 16890 tablet_bootstrap.cc:492] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:20.174881 16939 tablet_bootstrap.cc:492] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:20.176381 16890 tablet_bootstrap.cc:654] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.176977 16939 ts_tablet_manager.cc:1367] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20240502 08:18:20.177990 16939 raft_consensus.cc:348] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:20.179163 16939 raft_consensus.cc:374] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.179263 16939 raft_consensus.cc:729] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.179478 16939 consensus_queue.cc:260] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:20.180074 16939 ts_tablet_manager.cc:1398] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:20.180531 16945 tablet_bootstrap.cc:492] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:20.181563 16936 tablet.cc:1722] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:20.182027 16936 tablet.cc:1620] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:20.182139 16935 tablet.cc:1722] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:20.182564 16935 tablet.cc:1620] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:20.183297 16937 tablet.cc:1722] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:20.183701 16937 tablet.cc:1620] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:20.191624 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=121220bb856b46f39c335e87a2aee3c7] alter complete (version 2)
I20240502 08:18:20.186154 16945 tablet_bootstrap.cc:654] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.197477 16939 tablet_bootstrap.cc:492] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:20.198115 16944 tablet_bootstrap.cc:492] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:20.202093 16944 tablet_bootstrap.cc:654] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.205442 16939 tablet_bootstrap.cc:654] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.210707 16939 tablet_bootstrap.cc:492] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:20.210894 16945 tablet_bootstrap.cc:492] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:20.210901 16939 ts_tablet_manager.cc:1367] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 08:18:20.210988 16944 tablet_bootstrap.cc:492] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:20.210999 16890 tablet_bootstrap.cc:492] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:20.211150 16944 ts_tablet_manager.cc:1367] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 08:18:20.211400 16890 ts_tablet_manager.cc:1367] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.005s
I20240502 08:18:20.211616 16939 raft_consensus.cc:348] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:20.211875 16939 raft_consensus.cc:374] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.211840 16944 raft_consensus.cc:348] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:20.211979 16939 raft_consensus.cc:729] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.212172 16945 ts_tablet_manager.cc:1367] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.003s
I20240502 08:18:20.212260 16939 consensus_queue.cc:260] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:20.212349 16890 raft_consensus.cc:348] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:20.212648 16890 raft_consensus.cc:374] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.212765 16890 raft_consensus.cc:729] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.212826 16939 ts_tablet_manager.cc:1398] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:20.212934 16945 raft_consensus.cc:348] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:20.213164 16944 raft_consensus.cc:374] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.213059 16890 consensus_queue.cc:260] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:20.213272 16944 raft_consensus.cc:729] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.213375 16945 raft_consensus.cc:374] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.213677 16890 ts_tablet_manager.cc:1398] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:20.213706 16944 consensus_queue.cc:260] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:20.214217 16944 ts_tablet_manager.cc:1398] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:20.214408 16945 raft_consensus.cc:729] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.214685 16945 consensus_queue.cc:260] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:20.216871 16945 ts_tablet_manager.cc:1398] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 08:18:20.217581 16945 tablet_bootstrap.cc:492] T 438591418b6044f0abbe4d56bba217a2 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:20.222841 16938 tablet_bootstrap.cc:492] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:20.223119 16945 tablet_bootstrap.cc:654] T 438591418b6044f0abbe4d56bba217a2 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.224439 16938 tablet_bootstrap.cc:654] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.226794 16945 tablet_bootstrap.cc:492] T 438591418b6044f0abbe4d56bba217a2 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:20.226995 16945 ts_tablet_manager.cc:1367] T 438591418b6044f0abbe4d56bba217a2 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240502 08:18:20.227849 16945 raft_consensus.cc:348] T 438591418b6044f0abbe4d56bba217a2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:20.228128 16945 raft_consensus.cc:374] T 438591418b6044f0abbe4d56bba217a2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.228238 16945 raft_consensus.cc:729] T 438591418b6044f0abbe4d56bba217a2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.228536 16945 consensus_queue.cc:260] T 438591418b6044f0abbe4d56bba217a2 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:20.229314 16940 tablet_bootstrap.cc:492] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:20.229578 16940 ts_tablet_manager.cc:1367] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.075s user 0.012s sys 0.025s
I20240502 08:18:20.230866 16945 ts_tablet_manager.cc:1398] T 438591418b6044f0abbe4d56bba217a2 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:20.232115 16940 raft_consensus.cc:348] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:20.232383 16940 raft_consensus.cc:374] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.232507 16940 raft_consensus.cc:729] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.232748 16938 tablet_bootstrap.cc:492] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:20.232789 16940 consensus_queue.cc:260] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:20.232933 16938 ts_tablet_manager.cc:1367] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20240502 08:18:20.233536 16926 raft_consensus.cc:482] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:20.233668 16926 raft_consensus.cc:504] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:20.234365 16926 leader_election.cc:288] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:20.234818 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438591418b6044f0abbe4d56bba217a2" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:20.235217 12992 raft_consensus.cc:2454] T 438591418b6044f0abbe4d56bba217a2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:20.235729 13056 leader_election.cc:302] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:20.235895 16940 ts_tablet_manager.cc:1398] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 08:18:20.236038 16926 raft_consensus.cc:2781] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:20.236157 16926 raft_consensus.cc:482] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:20.236246 16926 raft_consensus.cc:3037] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.242074 16926 raft_consensus.cc:504] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:20.243007 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438591418b6044f0abbe4d56bba217a2" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:20.243223 12992 raft_consensus.cc:3037] T 438591418b6044f0abbe4d56bba217a2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.246093 12992 raft_consensus.cc:2454] T 438591418b6044f0abbe4d56bba217a2 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:20.246610 13056 leader_election.cc:302] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:20.246923 16835 raft_consensus.cc:2781] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:20.249006 16926 leader_election.cc:288] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:20.250968 16938 raft_consensus.cc:348] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:20.251168 16938 raft_consensus.cc:374] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.251256 16938 raft_consensus.cc:729] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.251462 16938 consensus_queue.cc:260] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:20.252084 16938 ts_tablet_manager.cc:1398] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240502 08:18:20.252388 16835 raft_consensus.cc:686] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:18:20.253001 16835 consensus_queue.cc:237] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:20.260145 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438591418b6044f0abbe4d56bba217a2" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:20.260337 15755 raft_consensus.cc:2454] T 438591418b6044f0abbe4d56bba217a2 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:20.260643 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "438591418b6044f0abbe4d56bba217a2" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:20.260828 15754 raft_consensus.cc:3037] T 438591418b6044f0abbe4d56bba217a2 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.262737 15754 raft_consensus.cc:2454] T 438591418b6044f0abbe4d56bba217a2 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:20.263820 12587 catalog_manager.cc:5484] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:20.297133 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:20.304200 16935 tablet.cc:1722] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:20.305045 16937 tablet.cc:1722] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:20.305979 16936 tablet.cc:1722] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:20.313967 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=121220bb856b46f39c335e87a2aee3c7] alter complete (version 3)
I20240502 08:18:20.392123 16908 raft_consensus.cc:482] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:20.392279 16908 raft_consensus.cc:504] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:20.393008 16908 leader_election.cc:288] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:20.393391 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6a660632634e3c9abde21dbf84dfc0" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:20.393466 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6a660632634e3c9abde21dbf84dfc0" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:20.393599 15886 raft_consensus.cc:2454] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:20.393673 12727 raft_consensus.cc:2454] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:20.394028 15555 leader_election.cc:302] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:20.394305 16908 raft_consensus.cc:2781] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:20.394416 16908 raft_consensus.cc:482] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:20.394524 16908 raft_consensus.cc:3037] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.396389 16908 raft_consensus.cc:504] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:20.396899 16908 leader_election.cc:288] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:20.397218 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6a660632634e3c9abde21dbf84dfc0" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:20.397420 15886 raft_consensus.cc:3037] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.397392 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6a660632634e3c9abde21dbf84dfc0" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:20.397562 12727 raft_consensus.cc:3037] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.399207 15886 raft_consensus.cc:2454] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:20.399454 12727 raft_consensus.cc:2454] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:20.399534 15555 leader_election.cc:302] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:20.399837 16908 raft_consensus.cc:2781] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:20.399978 16908 raft_consensus.cc:686] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:20.400239 16908 consensus_queue.cc:237] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:20.403249 12584 catalog_manager.cc:5484] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:20.415791 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:20.416007 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:20.418617 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet ab14ea85b65e47c1b857a4250407980f
I20240502 08:18:20.419275 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet ab14ea85b65e47c1b857a4250407980f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:20 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:20.419373 15734 tablet_service.cc:1514] Processing DeleteTablet for tablet ab14ea85b65e47c1b857a4250407980f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:20 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:20.419517 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet ab14ea85b65e47c1b857a4250407980f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:20 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:20.420171 16958 tablet_replica.cc:330] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:20.420400 16958 raft_consensus.cc:2229] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:20.420615 16958 raft_consensus.cc:2258] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:20.421069 16957 tablet_replica.cc:330] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:20.421247 16957 raft_consensus.cc:2229] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:20.421433 16957 raft_consensus.cc:2258] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:20.421710 16958 ts_tablet_manager.cc:1875] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:20.421954 16959 tablet_replica.cc:330] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:20.422154 16959 raft_consensus.cc:2229] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:20.422276 16957 ts_tablet_manager.cc:1875] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:20.422552 16959 raft_consensus.cc:2258] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:20.423390 16959 ts_tablet_manager.cc:1875] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:20.428411 16957 ts_tablet_manager.cc:1888] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:20.428571 16957 log.cc:1196] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/ab14ea85b65e47c1b857a4250407980f
I20240502 08:18:20.429055 16957 ts_tablet_manager.cc:1909] T ab14ea85b65e47c1b857a4250407980f P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:20.429111 16958 ts_tablet_manager.cc:1888] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:20.429247 16958 log.cc:1196] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/ab14ea85b65e47c1b857a4250407980f
I20240502 08:18:20.429435 16959 ts_tablet_manager.cc:1888] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:20.429603 16959 log.cc:1196] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/ab14ea85b65e47c1b857a4250407980f
I20240502 08:18:20.429677 16958 ts_tablet_manager.cc:1909] T ab14ea85b65e47c1b857a4250407980f P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:20.430042 16959 ts_tablet_manager.cc:1909] T ab14ea85b65e47c1b857a4250407980f P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:20.430136 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet ab14ea85b65e47c1b857a4250407980f (table rebalancer_dynamic_table_new_name [id=121220bb856b46f39c335e87a2aee3c7]) successfully deleted
I20240502 08:18:20.430917 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet ab14ea85b65e47c1b857a4250407980f (table rebalancer_dynamic_table_new_name [id=121220bb856b46f39c335e87a2aee3c7]) successfully deleted
I20240502 08:18:20.431205 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet ab14ea85b65e47c1b857a4250407980f (table rebalancer_dynamic_table_new_name [id=121220bb856b46f39c335e87a2aee3c7]) successfully deleted
I20240502 08:18:20.495559 16734 raft_consensus.cc:482] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:20.495721 16734 raft_consensus.cc:504] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:20.496439 16734 leader_election.cc:288] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:20.496707 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ecdc518ec94e43b0867a0786575304" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:20.496943 15621 raft_consensus.cc:2454] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:20.496901 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ecdc518ec94e43b0867a0786575304" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:20.497115 15487 raft_consensus.cc:2454] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:20.497285 12928 leader_election.cc:302] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:20.497627 16734 raft_consensus.cc:2781] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:20.497722 16734 raft_consensus.cc:482] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:20.497799 16734 raft_consensus.cc:3037] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.499835 16734 raft_consensus.cc:504] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:20.500303 16734 leader_election.cc:288] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:20.500718 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ecdc518ec94e43b0867a0786575304" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:20.500936 15621 raft_consensus.cc:3037] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.500895 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ecdc518ec94e43b0867a0786575304" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:20.501081 15487 raft_consensus.cc:3037] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.503577 15621 raft_consensus.cc:2454] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:20.503667 15487 raft_consensus.cc:2454] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:20.504005 12928 leader_election.cc:302] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:20.504272 16734 raft_consensus.cc:2781] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:20.504483 16734 raft_consensus.cc:686] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:20.504757 16734 consensus_queue.cc:237] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:20.508014 12584 catalog_manager.cc:5484] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:20.520936 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:20.527973 15734 tablet_service.cc:1467] Processing CreateTablet for tablet dbd0eaf9ba5140639f844fa93ac3196a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3e031a36d9874a38ae0874c6ceb82dda]), partition=
I20240502 08:18:20.528328 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd0eaf9ba5140639f844fa93ac3196a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.528379 15467 tablet_service.cc:1467] Processing CreateTablet for tablet dbd0eaf9ba5140639f844fa93ac3196a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3e031a36d9874a38ae0874c6ceb82dda]), partition=
I20240502 08:18:20.528434 15865 tablet_service.cc:1467] Processing CreateTablet for tablet dbd0eaf9ba5140639f844fa93ac3196a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3e031a36d9874a38ae0874c6ceb82dda]), partition=
I20240502 08:18:20.528672 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd0eaf9ba5140639f844fa93ac3196a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.528748 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd0eaf9ba5140639f844fa93ac3196a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:20.533722 16890 tablet_bootstrap.cc:492] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:20.535133 16890 tablet_bootstrap.cc:654] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.535699 16938 tablet_bootstrap.cc:492] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:20.536713 16887 tablet_bootstrap.cc:492] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:20.536931 16938 tablet_bootstrap.cc:654] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.538136 16887 tablet_bootstrap.cc:654] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:20.539222 16938 tablet_bootstrap.cc:492] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:20.539398 16938 ts_tablet_manager.cc:1367] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:20.539485 16890 tablet_bootstrap.cc:492] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:20.539633 16890 ts_tablet_manager.cc:1367] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:18:20.540205 16938 raft_consensus.cc:348] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:20.540443 16938 raft_consensus.cc:374] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.540524 16938 raft_consensus.cc:729] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.540536 16890 raft_consensus.cc:348] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:20.540786 16890 raft_consensus.cc:374] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.540874 16890 raft_consensus.cc:729] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.540752 16938 consensus_queue.cc:260] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:20.541100 16890 consensus_queue.cc:260] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:20.541332 16938 ts_tablet_manager.cc:1398] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:20.543882 16890 ts_tablet_manager.cc:1398] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:20.548947 16887 tablet_bootstrap.cc:492] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:20.549093 16887 ts_tablet_manager.cc:1367] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240502 08:18:20.549708 16887 raft_consensus.cc:348] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:20.549891 16887 raft_consensus.cc:374] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:20.549988 16887 raft_consensus.cc:729] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:20.550194 16887 consensus_queue.cc:260] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:20.550964 16887 ts_tablet_manager.cc:1398] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:20.771275 16835 consensus_queue.cc:1035] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.775621 16964 consensus_queue.cc:1035] T 438591418b6044f0abbe4d56bba217a2 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.832310 16915 raft_consensus.cc:482] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:20.832510 16915 raft_consensus.cc:504] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:20.833309 16915 leader_election.cc:288] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:20.833654 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd0eaf9ba5140639f844fa93ac3196a" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:20.833705 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd0eaf9ba5140639f844fa93ac3196a" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:20.833868 15754 raft_consensus.cc:2454] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:20.833870 15886 raft_consensus.cc:2454] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:20.834265 15420 leader_election.cc:302] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:20.834560 16915 raft_consensus.cc:2781] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:20.834662 16915 raft_consensus.cc:482] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:20.834748 16915 raft_consensus.cc:3037] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.836597 16915 raft_consensus.cc:504] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:20.837091 16915 leader_election.cc:288] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:20.837373 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd0eaf9ba5140639f844fa93ac3196a" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:20.837543 15754 raft_consensus.cc:3037] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.837534 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd0eaf9ba5140639f844fa93ac3196a" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:20.837697 15886 raft_consensus.cc:3037] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:20.839385 15754 raft_consensus.cc:2454] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:20.839473 15886 raft_consensus.cc:2454] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:20.839694 15418 leader_election.cc:302] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:20.839988 16915 raft_consensus.cc:2781] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:20.840130 16915 raft_consensus.cc:686] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:20.840396 16915 consensus_queue.cc:237] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:20.841713 16909 consensus_queue.cc:1035] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:20.845412 12584 catalog_manager.cc:5484] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:20.848227 16909 consensus_queue.cc:1035] T 4e6a660632634e3c9abde21dbf84dfc0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.004863 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:21.010699 15754 raft_consensus.cc:1261] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:21.010757 15886 raft_consensus.cc:1261] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:21.011338 16777 consensus_queue.cc:1035] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.012259 16777 consensus_queue.cc:1035] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.015648 12583 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet dbd0eaf9ba5140639f844fa93ac3196a (table rebalancer_dynamic_table [id=3e031a36d9874a38ae0874c6ceb82dda]). Expected version 1 got 0
I20240502 08:18:21.026168 16979 tablet.cc:1722] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:21.026940 16979 tablet.cc:1620] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:21.028688 12584 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet dbd0eaf9ba5140639f844fa93ac3196a (table rebalancer_dynamic_table [id=3e031a36d9874a38ae0874c6ceb82dda]). Expected version 1 got 0
I20240502 08:18:21.036680 16980 tablet.cc:1722] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:21.037170 16980 tablet.cc:1620] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:21.036944 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:21.041297 12587 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet dbd0eaf9ba5140639f844fa93ac3196a (table rebalancer_dynamic_table [id=3e031a36d9874a38ae0874c6ceb82dda]). Expected version 1 got 0
I20240502 08:18:21.050012 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 3b930dcb3863402296a205fdb65d6fc3 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=863bf9b609384908ae899e09f3d2e04c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.050498 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b930dcb3863402296a205fdb65d6fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.050603 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 7082603ad2d6450e8863cd53b1897b2e (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=863bf9b609384908ae899e09f3d2e04c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.050989 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7082603ad2d6450e8863cd53b1897b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.051396 16889 consensus_queue.cc:1035] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.055420 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 3b930dcb3863402296a205fdb65d6fc3 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=863bf9b609384908ae899e09f3d2e04c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.055850 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b930dcb3863402296a205fdb65d6fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.055969 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 7082603ad2d6450e8863cd53b1897b2e (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=863bf9b609384908ae899e09f3d2e04c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.056304 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7082603ad2d6450e8863cd53b1897b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.059810 16983 tablet_bootstrap.cc:492] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:21.060241 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 3b930dcb3863402296a205fdb65d6fc3 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=863bf9b609384908ae899e09f3d2e04c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.060837 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b930dcb3863402296a205fdb65d6fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.062399 16983 tablet_bootstrap.cc:654] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.067682 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 7082603ad2d6450e8863cd53b1897b2e (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=863bf9b609384908ae899e09f3d2e04c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.068274 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7082603ad2d6450e8863cd53b1897b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.068704 15866 tablet_service.cc:1467] Processing CreateTablet for tablet 4480db72a1104bdc9b0700303b40f1b8 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=863bf9b609384908ae899e09f3d2e04c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.069245 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4480db72a1104bdc9b0700303b40f1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.073079 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 4480db72a1104bdc9b0700303b40f1b8 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=863bf9b609384908ae899e09f3d2e04c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.073565 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4480db72a1104bdc9b0700303b40f1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.078500 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=3e031a36d9874a38ae0874c6ceb82dda] alter complete (version 1)
I20240502 08:18:21.082780 16987 tablet_bootstrap.cc:492] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:21.083367 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 4480db72a1104bdc9b0700303b40f1b8 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=863bf9b609384908ae899e09f3d2e04c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.083726 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4480db72a1104bdc9b0700303b40f1b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.084492 16985 tablet_bootstrap.cc:492] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:21.088907 16985 tablet_bootstrap.cc:654] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.094130 16987 tablet_bootstrap.cc:654] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.096827 16981 tablet.cc:1722] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:21.099571 16981 tablet.cc:1620] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:21.104058 12587 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet dbd0eaf9ba5140639f844fa93ac3196a (table rebalancer_dynamic_table [id=3e031a36d9874a38ae0874c6ceb82dda]). Expected version 1 got 0
I20240502 08:18:21.108788 16986 tablet_bootstrap.cc:492] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:21.110683 16988 tablet_bootstrap.cc:492] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:21.112425 16988 tablet_bootstrap.cc:654] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.115518 16986 tablet_bootstrap.cc:654] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.119355 16889 consensus_queue.cc:1035] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.119459 16991 tablet_bootstrap.cc:492] T 4480db72a1104bdc9b0700303b40f1b8 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:21.122890 16985 tablet_bootstrap.cc:492] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:21.123800 16985 ts_tablet_manager.cc:1367] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.000s
I20240502 08:18:21.124583 16985 raft_consensus.cc:348] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:21.124739 16985 raft_consensus.cc:374] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.124840 16985 raft_consensus.cc:729] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.125020 16985 consensus_queue.cc:260] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:21.126109 16985 ts_tablet_manager.cc:1398] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:21.128034 16991 tablet_bootstrap.cc:654] T 4480db72a1104bdc9b0700303b40f1b8 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.129375 16987 tablet_bootstrap.cc:492] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:21.129583 16987 ts_tablet_manager.cc:1367] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.003s
I20240502 08:18:21.130443 16987 raft_consensus.cc:348] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:21.130718 16987 raft_consensus.cc:374] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.130863 16991 tablet_bootstrap.cc:492] T 4480db72a1104bdc9b0700303b40f1b8 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:21.130857 16987 raft_consensus.cc:729] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.131083 16991 ts_tablet_manager.cc:1367] T 4480db72a1104bdc9b0700303b40f1b8 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240502 08:18:21.131107 16987 consensus_queue.cc:260] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:21.131934 16987 ts_tablet_manager.cc:1398] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:21.132238 16987 tablet_bootstrap.cc:492] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:21.135780 16987 tablet_bootstrap.cc:654] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.136821 16991 raft_consensus.cc:348] T 4480db72a1104bdc9b0700303b40f1b8 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:21.137488 16983 tablet_bootstrap.cc:492] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:21.137687 16983 ts_tablet_manager.cc:1367] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.078s user 0.020s sys 0.021s
I20240502 08:18:21.137750 16991 raft_consensus.cc:374] T 4480db72a1104bdc9b0700303b40f1b8 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.137925 16991 raft_consensus.cc:729] T 4480db72a1104bdc9b0700303b40f1b8 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.138247 16991 consensus_queue.cc:260] T 4480db72a1104bdc9b0700303b40f1b8 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:21.138448 16983 raft_consensus.cc:348] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:21.138744 16983 raft_consensus.cc:374] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.138885 16983 raft_consensus.cc:729] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.139179 16983 consensus_queue.cc:260] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:21.140311 16991 ts_tablet_manager.cc:1398] T 4480db72a1104bdc9b0700303b40f1b8 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 08:18:21.140667 16983 ts_tablet_manager.cc:1398] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:21.140939 16983 tablet_bootstrap.cc:492] T 4480db72a1104bdc9b0700303b40f1b8 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:21.142441 16983 tablet_bootstrap.cc:654] T 4480db72a1104bdc9b0700303b40f1b8 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.146649 16986 tablet_bootstrap.cc:492] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:21.146855 16986 ts_tablet_manager.cc:1367] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240502 08:18:21.147342 16983 tablet_bootstrap.cc:492] T 4480db72a1104bdc9b0700303b40f1b8 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:21.147747 16983 ts_tablet_manager.cc:1367] T 4480db72a1104bdc9b0700303b40f1b8 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240502 08:18:21.148469 16983 raft_consensus.cc:348] T 4480db72a1104bdc9b0700303b40f1b8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:21.148686 16983 raft_consensus.cc:374] T 4480db72a1104bdc9b0700303b40f1b8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.148777 16983 raft_consensus.cc:729] T 4480db72a1104bdc9b0700303b40f1b8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.148998 16983 consensus_queue.cc:260] T 4480db72a1104bdc9b0700303b40f1b8 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:21.149420 16777 raft_consensus.cc:482] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:21.149583 16777 raft_consensus.cc:504] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:21.149607 16986 raft_consensus.cc:348] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:21.149830 16986 raft_consensus.cc:374] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.149909 16986 raft_consensus.cc:729] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.150106 16986 consensus_queue.cc:260] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:21.150282 16777 leader_election.cc:288] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:21.150843 16986 ts_tablet_manager.cc:1398] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:21.151114 16986 tablet_bootstrap.cc:492] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:21.151047 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b930dcb3863402296a205fdb65d6fc3" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:21.151764 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b930dcb3863402296a205fdb65d6fc3" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:21.151998 12859 raft_consensus.cc:2454] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:21.153582 16983 ts_tablet_manager.cc:1398] T 4480db72a1104bdc9b0700303b40f1b8 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20240502 08:18:21.153558 15418 leader_election.cc:341] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:21.154116 15419 leader_election.cc:302] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db7893bb29a047b3b82cc60bfd8930a3, e1dd14f7ca764a1490df419f13f2804a; no voters: 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:18:21.154619 16777 raft_consensus.cc:2781] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:21.154738 16777 raft_consensus.cc:482] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:21.154830 16777 raft_consensus.cc:3037] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.156992 16986 tablet_bootstrap.cc:654] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.160861 16777 raft_consensus.cc:504] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:21.161561 16988 tablet_bootstrap.cc:492] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:21.161756 16988 ts_tablet_manager.cc:1367] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.051s user 0.015s sys 0.019s
I20240502 08:18:21.161898 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b930dcb3863402296a205fdb65d6fc3" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
W20240502 08:18:21.162396 15418 leader_election.cc:341] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:21.162721 16988 raft_consensus.cc:348] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:21.162974 16988 raft_consensus.cc:374] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.162994 16987 tablet_bootstrap.cc:492] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:21.163132 16988 raft_consensus.cc:729] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.163151 16987 ts_tablet_manager.cc:1367] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.008s
I20240502 08:18:21.163209 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b930dcb3863402296a205fdb65d6fc3" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:21.163429 12859 raft_consensus.cc:3037] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.163472 16988 consensus_queue.cc:260] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:21.163918 16987 raft_consensus.cc:348] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:21.164214 16987 raft_consensus.cc:374] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.164330 16987 raft_consensus.cc:729] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.164634 16987 consensus_queue.cc:260] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:21.165223 16987 ts_tablet_manager.cc:1398] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:21.166116 12859 raft_consensus.cc:2454] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:21.166815 16988 ts_tablet_manager.cc:1398] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:21.167703 15419 leader_election.cc:302] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db7893bb29a047b3b82cc60bfd8930a3, e1dd14f7ca764a1490df419f13f2804a; no voters: 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:18:21.167980 16915 raft_consensus.cc:2781] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:21.169432 16777 leader_election.cc:288] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:21.169593 16986 tablet_bootstrap.cc:492] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:21.169975 16915 raft_consensus.cc:686] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:21.170231 16915 consensus_queue.cc:237] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:21.169808 16986 ts_tablet_manager.cc:1367] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20240502 08:18:21.172479 16986 raft_consensus.cc:348] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:21.172698 16986 raft_consensus.cc:374] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.172783 16986 raft_consensus.cc:729] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.173038 16986 consensus_queue.cc:260] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:21.173606 16986 ts_tablet_manager.cc:1398] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:21.173936 12587 catalog_manager.cc:5484] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:21.186376 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:21.193450 16979 tablet.cc:1722] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:21.193984 16979 tablet.cc:1620] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:21.194053 16981 tablet.cc:1722] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:21.194437 16981 tablet.cc:1620] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:21.194626 16980 tablet.cc:1722] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:21.195035 16980 tablet.cc:1620] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:21.205473 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=3e031a36d9874a38ae0874c6ceb82dda] alter complete (version 2)
I20240502 08:18:21.306577 16777 raft_consensus.cc:482] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:21.306754 16777 raft_consensus.cc:504] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:21.307466 16777 leader_election.cc:288] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:21.307780 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4480db72a1104bdc9b0700303b40f1b8" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:21.308007 15754 raft_consensus.cc:2454] T 4480db72a1104bdc9b0700303b40f1b8 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:21.308027 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4480db72a1104bdc9b0700303b40f1b8" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:21.308211 15887 raft_consensus.cc:2454] T 4480db72a1104bdc9b0700303b40f1b8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:21.308400 15418 leader_election.cc:302] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:21.308672 16777 raft_consensus.cc:2781] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:21.308784 16777 raft_consensus.cc:482] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:21.308868 16777 raft_consensus.cc:3037] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.310761 16777 raft_consensus.cc:504] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:21.311275 16777 leader_election.cc:288] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:21.311676 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4480db72a1104bdc9b0700303b40f1b8" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:21.311722 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4480db72a1104bdc9b0700303b40f1b8" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:21.311884 15887 raft_consensus.cc:3037] T 4480db72a1104bdc9b0700303b40f1b8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.311908 15754 raft_consensus.cc:3037] T 4480db72a1104bdc9b0700303b40f1b8 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.312110 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:21.314636 15887 raft_consensus.cc:2454] T 4480db72a1104bdc9b0700303b40f1b8 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:21.314872 15754 raft_consensus.cc:2454] T 4480db72a1104bdc9b0700303b40f1b8 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:21.315023 15420 leader_election.cc:302] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:21.315351 16777 raft_consensus.cc:2781] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:21.315523 16777 raft_consensus.cc:686] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:21.315822 16777 consensus_queue.cc:237] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:21.323098 16980 tablet.cc:1722] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:21.323608 16979 tablet.cc:1722] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:21.325327 12579 catalog_manager.cc:5484] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:21.329501 16964 raft_consensus.cc:482] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:21.329677 16964 raft_consensus.cc:504] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:21.330413 16964 leader_election.cc:288] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:21.332592 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b930dcb3863402296a205fdb65d6fc3" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:21.332868 12859 raft_consensus.cc:2379] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a0b05f3250d415590b0edf6b44fbe1f in current term 1: Already voted for candidate db7893bb29a047b3b82cc60bfd8930a3 in this term.
I20240502 08:18:21.335602 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=3e031a36d9874a38ae0874c6ceb82dda] alter complete (version 3)
I20240502 08:18:21.352030 16981 tablet.cc:1722] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:21.358966 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b930dcb3863402296a205fdb65d6fc3" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:21.359515 13056 leader_election.cc:302] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f; no voters: db7893bb29a047b3b82cc60bfd8930a3, e1dd14f7ca764a1490df419f13f2804a
I20240502 08:18:21.359942 16964 raft_consensus.cc:3037] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.362694 16964 raft_consensus.cc:2726] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:21.429499 16909 raft_consensus.cc:482] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:21.429672 16909 raft_consensus.cc:504] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:21.430346 16909 leader_election.cc:288] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:21.430711 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7082603ad2d6450e8863cd53b1897b2e" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:21.430943 13119 raft_consensus.cc:2454] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:21.430913 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7082603ad2d6450e8863cd53b1897b2e" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:21.431161 15887 raft_consensus.cc:2454] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:21.431411 15553 leader_election.cc:302] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:21.431686 16909 raft_consensus.cc:2781] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:21.431771 16909 raft_consensus.cc:482] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:21.431866 16909 raft_consensus.cc:3037] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.433712 16909 raft_consensus.cc:504] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:21.434214 16909 leader_election.cc:288] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:21.434538 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7082603ad2d6450e8863cd53b1897b2e" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:21.434587 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7082603ad2d6450e8863cd53b1897b2e" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:21.434746 13119 raft_consensus.cc:3037] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.434764 15887 raft_consensus.cc:3037] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.436940 15887 raft_consensus.cc:2454] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:21.437372 15555 leader_election.cc:302] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:21.437515 13119 raft_consensus.cc:2454] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:21.437577 16909 raft_consensus.cc:2781] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:21.438019 16909 raft_consensus.cc:686] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:21.438426 16909 consensus_queue.cc:237] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:21.441224 12583 catalog_manager.cc:5484] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:21.445823 12583 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:21.446022 12583 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:21.448937 12583 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet dbd0eaf9ba5140639f844fa93ac3196a
I20240502 08:18:21.449712 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd0eaf9ba5140639f844fa93ac3196a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:21 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:21.449724 15734 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd0eaf9ba5140639f844fa93ac3196a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:21 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:21.450594 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd0eaf9ba5140639f844fa93ac3196a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:21 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:21.452546 17011 tablet_replica.cc:330] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:21.452786 17011 raft_consensus.cc:2229] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:21.450389 17009 tablet_replica.cc:330] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:21.453176 17011 raft_consensus.cc:2258] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:21.453307 17009 raft_consensus.cc:2229] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:21.453549 17009 raft_consensus.cc:2258] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:21.454200 17009 ts_tablet_manager.cc:1875] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:21.458707 17011 ts_tablet_manager.cc:1875] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:21.459441 17009 ts_tablet_manager.cc:1888] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:21.459564 17009 log.cc:1196] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/dbd0eaf9ba5140639f844fa93ac3196a
I20240502 08:18:21.459913 17009 ts_tablet_manager.cc:1909] T dbd0eaf9ba5140639f844fa93ac3196a P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:21.460788 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet dbd0eaf9ba5140639f844fa93ac3196a (table rebalancer_dynamic_table_new_name [id=3e031a36d9874a38ae0874c6ceb82dda]) successfully deleted
I20240502 08:18:21.464401 17010 tablet_replica.cc:330] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:21.464773 17010 raft_consensus.cc:2229] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:21.465010 17010 raft_consensus.cc:2258] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:21.465765 17010 ts_tablet_manager.cc:1875] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:21.467017 17011 ts_tablet_manager.cc:1888] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:21.467137 17011 log.cc:1196] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/dbd0eaf9ba5140639f844fa93ac3196a
I20240502 08:18:21.467557 17011 ts_tablet_manager.cc:1909] T dbd0eaf9ba5140639f844fa93ac3196a P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:21.468513 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet dbd0eaf9ba5140639f844fa93ac3196a (table rebalancer_dynamic_table_new_name [id=3e031a36d9874a38ae0874c6ceb82dda]) successfully deleted
I20240502 08:18:21.470991 17010 ts_tablet_manager.cc:1888] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:21.471091 17010 log.cc:1196] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/dbd0eaf9ba5140639f844fa93ac3196a
I20240502 08:18:21.471416 17010 ts_tablet_manager.cc:1909] T dbd0eaf9ba5140639f844fa93ac3196a P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:21.472143 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet dbd0eaf9ba5140639f844fa93ac3196a (table rebalancer_dynamic_table_new_name [id=3e031a36d9874a38ae0874c6ceb82dda]) successfully deleted
I20240502 08:18:21.552107 12583 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:21.558580 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 4c592a7b2b564b779d8e2e39539adbfa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b7b7d454a3204819bca9c58372bcc883]), partition=
I20240502 08:18:21.558765 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 4c592a7b2b564b779d8e2e39539adbfa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b7b7d454a3204819bca9c58372bcc883]), partition=
I20240502 08:18:21.558884 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 4c592a7b2b564b779d8e2e39539adbfa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b7b7d454a3204819bca9c58372bcc883]), partition=
I20240502 08:18:21.558981 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c592a7b2b564b779d8e2e39539adbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.559087 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c592a7b2b564b779d8e2e39539adbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.559245 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c592a7b2b564b779d8e2e39539adbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.564392 17012 tablet_bootstrap.cc:492] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:21.565025 16985 tablet_bootstrap.cc:492] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:21.565407 17013 tablet_bootstrap.cc:492] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:21.566112 17012 tablet_bootstrap.cc:654] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.566375 16985 tablet_bootstrap.cc:654] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.567276 17013 tablet_bootstrap.cc:654] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.569022 17012 tablet_bootstrap.cc:492] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:21.569278 17012 ts_tablet_manager.cc:1367] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20240502 08:18:21.570887 17012 raft_consensus.cc:348] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:21.571246 17012 raft_consensus.cc:374] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.571480 17012 raft_consensus.cc:729] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.571861 17012 consensus_queue.cc:260] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:21.572734 17012 ts_tablet_manager.cc:1398] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:21.576342 16985 tablet_bootstrap.cc:492] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:21.576540 16985 ts_tablet_manager.cc:1367] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240502 08:18:21.577275 17013 tablet_bootstrap.cc:492] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:21.577442 17013 ts_tablet_manager.cc:1367] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240502 08:18:21.577534 16985 raft_consensus.cc:348] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:21.577819 16985 raft_consensus.cc:374] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.578020 16985 raft_consensus.cc:729] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.578341 17013 raft_consensus.cc:348] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:21.578354 16985 consensus_queue.cc:260] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:21.578658 17013 raft_consensus.cc:374] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.578775 17013 raft_consensus.cc:729] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.579162 17013 consensus_queue.cc:260] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:21.579707 17013 ts_tablet_manager.cc:1398] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:21.582798 16985 ts_tablet_manager.cc:1398] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:21.622524 16889 raft_consensus.cc:482] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:21.622682 16889 raft_consensus.cc:504] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:21.623473 16889 leader_election.cc:288] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:21.623828 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c592a7b2b564b779d8e2e39539adbfa" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:21.623970 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c592a7b2b564b779d8e2e39539adbfa" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:21.624073 12727 raft_consensus.cc:2454] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:21.624190 12859 raft_consensus.cc:2454] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:21.624534 12926 leader_election.cc:302] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:21.624804 16889 raft_consensus.cc:2781] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:21.624926 16889 raft_consensus.cc:482] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:21.625018 16889 raft_consensus.cc:3037] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.626950 16889 raft_consensus.cc:504] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:21.627797 16889 leader_election.cc:288] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:21.628566 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c592a7b2b564b779d8e2e39539adbfa" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:21.628752 12727 raft_consensus.cc:3037] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.628813 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c592a7b2b564b779d8e2e39539adbfa" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:21.628970 12859 raft_consensus.cc:3037] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:21.631449 12727 raft_consensus.cc:2454] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:21.631726 12859 raft_consensus.cc:2454] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:21.632108 12926 leader_election.cc:302] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:18:21.632366 16889 raft_consensus.cc:2781] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:21.632511 16889 raft_consensus.cc:686] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:21.632800 16889 consensus_queue.cc:237] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:21.635764 12583 catalog_manager.cc:5484] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:21.689334 16915 consensus_queue.cc:1035] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.693131 16915 consensus_queue.cc:1035] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.699975 16915 consensus_queue.cc:1035] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.703992 17003 consensus_queue.cc:1035] T 4480db72a1104bdc9b0700303b40f1b8 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.747653 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:21.754102 12861 raft_consensus.cc:1261] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:21.754120 12727 raft_consensus.cc:1261] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:21.754745 16889 consensus_queue.cc:1035] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.755050 16992 consensus_queue.cc:1035] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.759776 12579 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet 4c592a7b2b564b779d8e2e39539adbfa (table rebalancer_dynamic_table [id=b7b7d454a3204819bca9c58372bcc883]). Expected version 1 got 0
I20240502 08:18:21.762039 17030 tablet.cc:1722] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:21.763437 17030 tablet.cc:1620] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:21.769241 17031 tablet.cc:1722] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:21.770515 17031 tablet.cc:1620] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:21.771293 12579 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet 4c592a7b2b564b779d8e2e39539adbfa (table rebalancer_dynamic_table [id=b7b7d454a3204819bca9c58372bcc883]). Expected version 1 got 0
I20240502 08:18:21.774209 12584 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet 4c592a7b2b564b779d8e2e39539adbfa (table rebalancer_dynamic_table [id=b7b7d454a3204819bca9c58372bcc883]). Expected version 1 got 0
I20240502 08:18:21.776664 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=b7b7d454a3204819bca9c58372bcc883] alter complete (version 1)
I20240502 08:18:21.779062 17032 tablet.cc:1722] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:21.779718 17032 tablet.cc:1620] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:21.781095 12584 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet 4c592a7b2b564b779d8e2e39539adbfa (table rebalancer_dynamic_table [id=b7b7d454a3204819bca9c58372bcc883]). Expected version 1 got 0
I20240502 08:18:21.835034 17007 consensus_queue.cc:1035] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.841336 17007 consensus_queue.cc:1035] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:21.881415 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:21.888046 17030 tablet.cc:1722] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:21.888514 17030 tablet.cc:1620] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:21.888988 17032 tablet.cc:1722] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:21.889426 17032 tablet.cc:1620] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:21.893021 17031 tablet.cc:1722] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:21.893471 17031 tablet.cc:1620] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:21.898818 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=b7b7d454a3204819bca9c58372bcc883] alter complete (version 2)
I20240502 08:18:21.953269 12583 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:21.961596 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 9296dc82707f48f9ac729c2ce91529c0 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b9146919078d47a99126e22456f8f239]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.962047 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9296dc82707f48f9ac729c2ce91529c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.962093 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 9296dc82707f48f9ac729c2ce91529c0 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b9146919078d47a99126e22456f8f239]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.962536 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9296dc82707f48f9ac729c2ce91529c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.963385 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 0e882f58c7f344f7929b8a30bd65fa44 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b9146919078d47a99126e22456f8f239]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.963765 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e882f58c7f344f7929b8a30bd65fa44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.964115 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 0e882f58c7f344f7929b8a30bd65fa44 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b9146919078d47a99126e22456f8f239]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.964493 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e882f58c7f344f7929b8a30bd65fa44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.962126 15866 tablet_service.cc:1467] Processing CreateTablet for tablet 0e882f58c7f344f7929b8a30bd65fa44 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b9146919078d47a99126e22456f8f239]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.966821 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e882f58c7f344f7929b8a30bd65fa44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.966799 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5a4eb8278c45f1a8389397a8748070 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b9146919078d47a99126e22456f8f239]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.966814 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 9296dc82707f48f9ac729c2ce91529c0 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b9146919078d47a99126e22456f8f239]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.967135 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5a4eb8278c45f1a8389397a8748070. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.967137 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9296dc82707f48f9ac729c2ce91529c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.979000 17040 tablet_bootstrap.cc:492] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:21.979341 17039 tablet_bootstrap.cc:492] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:21.979404 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5a4eb8278c45f1a8389397a8748070 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b9146919078d47a99126e22456f8f239]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.979794 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5a4eb8278c45f1a8389397a8748070. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.979840 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5a4eb8278c45f1a8389397a8748070 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=b9146919078d47a99126e22456f8f239]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:21.980247 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5a4eb8278c45f1a8389397a8748070. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:21.980643 17040 tablet_bootstrap.cc:654] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.980785 17039 tablet_bootstrap.cc:654] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.985878 17012 tablet_bootstrap.cc:492] T 8d5a4eb8278c45f1a8389397a8748070 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:21.986405 17036 tablet_bootstrap.cc:492] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:21.987414 17012 tablet_bootstrap.cc:654] T 8d5a4eb8278c45f1a8389397a8748070 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.988094 17036 tablet_bootstrap.cc:654] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.990830 16985 tablet_bootstrap.cc:492] T 8d5a4eb8278c45f1a8389397a8748070 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:21.991634 17012 tablet_bootstrap.cc:492] T 8d5a4eb8278c45f1a8389397a8748070 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:21.991793 17012 ts_tablet_manager.cc:1367] T 8d5a4eb8278c45f1a8389397a8748070 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 08:18:21.992364 16985 tablet_bootstrap.cc:654] T 8d5a4eb8278c45f1a8389397a8748070 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:21.992522 17012 raft_consensus.cc:348] T 8d5a4eb8278c45f1a8389397a8748070 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:21.992785 17012 raft_consensus.cc:374] T 8d5a4eb8278c45f1a8389397a8748070 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.992864 17012 raft_consensus.cc:729] T 8d5a4eb8278c45f1a8389397a8748070 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:21.993108 17012 consensus_queue.cc:260] T 8d5a4eb8278c45f1a8389397a8748070 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:21.993628 17012 ts_tablet_manager.cc:1398] T 8d5a4eb8278c45f1a8389397a8748070 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:21.993911 17036 tablet_bootstrap.cc:492] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:21.994041 17036 ts_tablet_manager.cc:1367] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 08:18:21.994745 17036 raft_consensus.cc:348] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:21.996311 17039 tablet_bootstrap.cc:492] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:21.996526 17039 ts_tablet_manager.cc:1367] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.008s
I20240502 08:18:21.997880 17039 raft_consensus.cc:348] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:21.998173 17039 raft_consensus.cc:374] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:21.998302 17039 raft_consensus.cc:729] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.000945 17039 consensus_queue.cc:260] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.001283 16865 raft_consensus.cc:482] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:22.001400 16865 raft_consensus.cc:504] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.001791 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:22.002419 16865 leader_election.cc:288] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:22.002990 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e882f58c7f344f7929b8a30bd65fa44" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
W20240502 08:18:22.003625 15818 leader_election.cc:341] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:22.004117 16985 tablet_bootstrap.cc:492] T 8d5a4eb8278c45f1a8389397a8748070 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:22.004304 16985 ts_tablet_manager.cc:1367] T 8d5a4eb8278c45f1a8389397a8748070 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240502 08:18:22.004410 17039 ts_tablet_manager.cc:1398] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 08:18:22.004703 17039 tablet_bootstrap.cc:492] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:22.005110 16985 raft_consensus.cc:348] T 8d5a4eb8278c45f1a8389397a8748070 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.005384 16985 raft_consensus.cc:374] T 8d5a4eb8278c45f1a8389397a8748070 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.005514 16985 raft_consensus.cc:729] T 8d5a4eb8278c45f1a8389397a8748070 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.006186 17039 tablet_bootstrap.cc:654] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.005759 16985 consensus_queue.cc:260] T 8d5a4eb8278c45f1a8389397a8748070 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.006829 16985 ts_tablet_manager.cc:1398] T 8d5a4eb8278c45f1a8389397a8748070 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:22.007655 17038 tablet_bootstrap.cc:492] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:22.007675 17036 raft_consensus.cc:374] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.007769 17036 raft_consensus.cc:729] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.008044 17036 consensus_queue.cc:260] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.002831 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e882f58c7f344f7929b8a30bd65fa44" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:22.008386 15621 raft_consensus.cc:2454] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:22.008735 17036 ts_tablet_manager.cc:1398] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:18:22.009310 15822 leader_election.cc:302] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:22.009654 17038 tablet_bootstrap.cc:654] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.009876 16865 raft_consensus.cc:2781] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:22.010010 16865 raft_consensus.cc:482] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:22.010107 16865 raft_consensus.cc:3037] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.012625 16865 raft_consensus.cc:504] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.021759 16865 leader_election.cc:288] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:22.023032 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e882f58c7f344f7929b8a30bd65fa44" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:22.023237 17040 tablet_bootstrap.cc:492] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:22.023427 17040 ts_tablet_manager.cc:1367] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.020s
W20240502 08:18:22.023521 15818 leader_election.cc:341] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:22.024283 17040 raft_consensus.cc:348] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.024526 17040 raft_consensus.cc:374] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.024674 17040 raft_consensus.cc:729] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.024950 17040 consensus_queue.cc:260] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.025633 17040 ts_tablet_manager.cc:1398] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:22.025897 17030 tablet.cc:1722] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:22.027549 17040 tablet_bootstrap.cc:492] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:22.027596 17032 tablet.cc:1722] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:22.030639 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e882f58c7f344f7929b8a30bd65fa44" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:22.030846 15621 raft_consensus.cc:3037] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.031970 17036 tablet_bootstrap.cc:492] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:22.033511 17036 tablet_bootstrap.cc:654] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.033548 15621 raft_consensus.cc:2454] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:22.033689 17031 tablet.cc:1722] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:22.035071 15822 leader_election.cc:302] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:22.035490 16865 raft_consensus.cc:2781] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:22.035624 16865 raft_consensus.cc:686] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:22.035877 16865 consensus_queue.cc:237] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.040226 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=b7b7d454a3204819bca9c58372bcc883] alter complete (version 3)
I20240502 08:18:22.041682 17040 tablet_bootstrap.cc:654] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.042424 12586 catalog_manager.cc:5484] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:22.043344 17038 tablet_bootstrap.cc:492] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:22.043484 17038 ts_tablet_manager.cc:1367] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.000s
I20240502 08:18:22.045384 17038 raft_consensus.cc:348] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:22.045655 17038 raft_consensus.cc:374] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.045806 17038 raft_consensus.cc:729] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.046231 17038 consensus_queue.cc:260] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:22.048070 17038 ts_tablet_manager.cc:1398] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:22.049491 17039 tablet_bootstrap.cc:492] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:22.049698 17039 ts_tablet_manager.cc:1367] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.007s
I20240502 08:18:22.050660 17039 raft_consensus.cc:348] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:22.051146 17039 raft_consensus.cc:374] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.051276 17039 raft_consensus.cc:729] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.051738 17039 consensus_queue.cc:260] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:22.053246 17040 tablet_bootstrap.cc:492] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:22.053254 17036 tablet_bootstrap.cc:492] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:22.053400 17036 ts_tablet_manager.cc:1367] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240502 08:18:22.053445 17040 ts_tablet_manager.cc:1367] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.002s
I20240502 08:18:22.053623 17039 ts_tablet_manager.cc:1398] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:22.054214 17036 raft_consensus.cc:348] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:22.054214 17040 raft_consensus.cc:348] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.054417 17040 raft_consensus.cc:374] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.054553 17040 raft_consensus.cc:729] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.055364 17036 raft_consensus.cc:374] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.055467 17036 raft_consensus.cc:729] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.055716 17036 consensus_queue.cc:260] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:22.056143 17040 consensus_queue.cc:260] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.056787 17040 ts_tablet_manager.cc:1398] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:22.058079 17036 ts_tablet_manager.cc:1398] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:22.146546 12586 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:22.146857 12586 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:22.152261 12586 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 4c592a7b2b564b779d8e2e39539adbfa
I20240502 08:18:22.153057 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c592a7b2b564b779d8e2e39539adbfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:22 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:22.153287 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c592a7b2b564b779d8e2e39539adbfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:22 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:22.153380 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c592a7b2b564b779d8e2e39539adbfa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:22 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:22.164180 17051 tablet_replica.cc:330] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:22.164528 17051 raft_consensus.cc:2229] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:22.164916 17052 tablet_replica.cc:330] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:22.164836 17051 raft_consensus.cc:2258] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:22.165635 17052 raft_consensus.cc:2229] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:22.165870 17052 raft_consensus.cc:2258] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:22.166177 17051 ts_tablet_manager.cc:1875] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:22.167009 17052 ts_tablet_manager.cc:1875] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:22.169696 17053 tablet_replica.cc:330] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:22.169993 17053 raft_consensus.cc:2229] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:22.170343 17053 raft_consensus.cc:2258] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:22.171599 17053 ts_tablet_manager.cc:1875] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:22.172088 17051 ts_tablet_manager.cc:1888] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:22.172257 17051 log.cc:1196] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/4c592a7b2b564b779d8e2e39539adbfa
I20240502 08:18:22.172744 17051 ts_tablet_manager.cc:1909] T 4c592a7b2b564b779d8e2e39539adbfa P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:22.173843 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 4c592a7b2b564b779d8e2e39539adbfa (table rebalancer_dynamic_table_new_name [id=b7b7d454a3204819bca9c58372bcc883]) successfully deleted
I20240502 08:18:22.174319 17052 ts_tablet_manager.cc:1888] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:22.174489 17052 log.cc:1196] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/4c592a7b2b564b779d8e2e39539adbfa
I20240502 08:18:22.174906 17052 ts_tablet_manager.cc:1909] T 4c592a7b2b564b779d8e2e39539adbfa P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:22.175971 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 4c592a7b2b564b779d8e2e39539adbfa (table rebalancer_dynamic_table_new_name [id=b7b7d454a3204819bca9c58372bcc883]) successfully deleted
I20240502 08:18:22.178072 17053 ts_tablet_manager.cc:1888] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:22.178175 17053 log.cc:1196] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/4c592a7b2b564b779d8e2e39539adbfa
I20240502 08:18:22.178557 17053 ts_tablet_manager.cc:1909] T 4c592a7b2b564b779d8e2e39539adbfa P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:22.179303 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 4c592a7b2b564b779d8e2e39539adbfa (table rebalancer_dynamic_table_new_name [id=b7b7d454a3204819bca9c58372bcc883]) successfully deleted
I20240502 08:18:22.180450 16865 raft_consensus.cc:482] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:22.180617 16865 raft_consensus.cc:504] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:22.181658 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9296dc82707f48f9ac729c2ce91529c0" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:22.181918 15487 raft_consensus.cc:2454] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:22.182160 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9296dc82707f48f9ac729c2ce91529c0" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:22.182385 15621 raft_consensus.cc:2454] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:22.182556 15819 leader_election.cc:302] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:22.182863 16632 raft_consensus.cc:2781] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:22.182955 16632 raft_consensus.cc:482] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:22.183051 16632 raft_consensus.cc:3037] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.184775 16632 raft_consensus.cc:504] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:22.185312 16632 leader_election.cc:288] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:22.185576 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9296dc82707f48f9ac729c2ce91529c0" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:22.185760 15487 raft_consensus.cc:3037] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.186126 16865 leader_election.cc:288] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:22.186094 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9296dc82707f48f9ac729c2ce91529c0" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:22.186312 15621 raft_consensus.cc:3037] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.188076 15621 raft_consensus.cc:2454] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:22.188359 15487 raft_consensus.cc:2454] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:22.188467 15822 leader_election.cc:302] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:22.188843 17054 raft_consensus.cc:2781] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:22.189170 17054 raft_consensus.cc:686] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:22.189458 17054 consensus_queue.cc:237] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:22.192282 12586 catalog_manager.cc:5484] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:22.254418 12586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:22.261113 12842 tablet_service.cc:1467] Processing CreateTablet for tablet c0bfe5dc732047a89e3dc0a6fbe7a120 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a248e166677c45e39d658bec33c543f7]), partition=
I20240502 08:18:22.261423 15734 tablet_service.cc:1467] Processing CreateTablet for tablet c0bfe5dc732047a89e3dc0a6fbe7a120 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a248e166677c45e39d658bec33c543f7]), partition=
I20240502 08:18:22.261478 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bfe5dc732047a89e3dc0a6fbe7a120. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.261710 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bfe5dc732047a89e3dc0a6fbe7a120. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.261704 13101 tablet_service.cc:1467] Processing CreateTablet for tablet c0bfe5dc732047a89e3dc0a6fbe7a120 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a248e166677c45e39d658bec33c543f7]), partition=
I20240502 08:18:22.262035 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bfe5dc732047a89e3dc0a6fbe7a120. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.267926 16985 tablet_bootstrap.cc:492] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:22.268568 17040 tablet_bootstrap.cc:492] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:22.269513 16985 tablet_bootstrap.cc:654] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.269937 17040 tablet_bootstrap.cc:654] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.273450 17040 tablet_bootstrap.cc:492] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:22.273581 17040 ts_tablet_manager.cc:1367] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:18:22.274317 17040 raft_consensus.cc:348] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:22.274534 17040 raft_consensus.cc:374] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.274626 17040 raft_consensus.cc:729] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.274860 17040 consensus_queue.cc:260] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:22.275415 17040 ts_tablet_manager.cc:1398] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:22.277768 17058 tablet_bootstrap.cc:492] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:22.279347 17058 tablet_bootstrap.cc:654] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:22.279635 16985 tablet_bootstrap.cc:492] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:22.279786 16985 ts_tablet_manager.cc:1367] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 08:18:22.280532 16985 raft_consensus.cc:348] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:22.280771 16985 raft_consensus.cc:374] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.280854 16985 raft_consensus.cc:729] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.282354 16985 consensus_queue.cc:260] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:22.283013 16985 ts_tablet_manager.cc:1398] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:22.283684 17058 tablet_bootstrap.cc:492] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:22.283831 17058 ts_tablet_manager.cc:1367] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 08:18:22.284570 17058 raft_consensus.cc:348] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:22.284785 17058 raft_consensus.cc:374] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:22.284884 17058 raft_consensus.cc:729] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:22.285095 17058 consensus_queue.cc:260] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:22.285679 17058 ts_tablet_manager.cc:1398] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:22.365293 16854 raft_consensus.cc:482] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:22.365434 16854 raft_consensus.cc:504] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.366046 16854 leader_election.cc:288] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:22.366544 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5a4eb8278c45f1a8389397a8748070" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:22.366524 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5a4eb8278c45f1a8389397a8748070" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:22.366772 12990 raft_consensus.cc:2454] T 8d5a4eb8278c45f1a8389397a8748070 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:22.366781 12861 raft_consensus.cc:2454] T 8d5a4eb8278c45f1a8389397a8748070 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:22.367165 15688 leader_election.cc:302] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:22.367449 16854 raft_consensus.cc:2781] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:22.367551 16854 raft_consensus.cc:482] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:22.367636 16854 raft_consensus.cc:3037] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.369801 16854 raft_consensus.cc:504] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.370718 16854 leader_election.cc:288] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:22.371089 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5a4eb8278c45f1a8389397a8748070" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:22.371258 12992 raft_consensus.cc:3037] T 8d5a4eb8278c45f1a8389397a8748070 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.371225 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5a4eb8278c45f1a8389397a8748070" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:22.371428 12861 raft_consensus.cc:3037] T 8d5a4eb8278c45f1a8389397a8748070 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.373194 12992 raft_consensus.cc:2454] T 8d5a4eb8278c45f1a8389397a8748070 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:22.373587 15688 leader_election.cc:302] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:22.373823 12861 raft_consensus.cc:2454] T 8d5a4eb8278c45f1a8389397a8748070 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:22.373857 16854 raft_consensus.cc:2781] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:22.379642 16854 raft_consensus.cc:686] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:22.379925 16854 consensus_queue.cc:237] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.383394 12587 catalog_manager.cc:5484] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:22.402943 17025 raft_consensus.cc:482] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:22.403110 17025 raft_consensus.cc:504] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:22.403832 17025 leader_election.cc:288] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:22.404174 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bfe5dc732047a89e3dc0a6fbe7a120" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:22.404289 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bfe5dc732047a89e3dc0a6fbe7a120" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:22.404404 15754 raft_consensus.cc:2454] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:22.404511 13121 raft_consensus.cc:2454] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:22.404770 12795 leader_election.cc:302] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:22.405022 17025 raft_consensus.cc:2781] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:22.405157 17025 raft_consensus.cc:482] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:22.405261 17025 raft_consensus.cc:3037] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.407469 17025 raft_consensus.cc:504] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:22.407919 17025 leader_election.cc:288] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:22.408282 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bfe5dc732047a89e3dc0a6fbe7a120" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:22.408366 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bfe5dc732047a89e3dc0a6fbe7a120" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:22.408478 15754 raft_consensus.cc:3037] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.408566 13121 raft_consensus.cc:3037] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.411176 15754 raft_consensus.cc:2454] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:22.411198 13121 raft_consensus.cc:2454] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:22.411556 12795 leader_election.cc:302] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:22.411852 17025 raft_consensus.cc:2781] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:22.412652 17025 raft_consensus.cc:686] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:18:22.412943 17025 consensus_queue.cc:237] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:22.416288 12587 catalog_manager.cc:5484] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:22.422868 16854 raft_consensus.cc:482] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:22.423003 16854 raft_consensus.cc:504] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:22.423666 16854 leader_election.cc:288] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:22.424082 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e882f58c7f344f7929b8a30bd65fa44" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:22.424154 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e882f58c7f344f7929b8a30bd65fa44" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:22.424414 15621 raft_consensus.cc:2379] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9333cb10227b4801b4fa7c24e22e131b in current term 1: Already voted for candidate 10f0e36203914b85b77e78609ffced98 in this term.
I20240502 08:18:22.424764 15690 leader_election.cc:302] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b; no voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:22.425071 16854 raft_consensus.cc:3037] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:22.426891 16854 raft_consensus.cc:2726] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:22.534689 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:22.540300 13121 raft_consensus.cc:1261] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:22.540416 15754 raft_consensus.cc:1261] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:22.541128 17062 consensus_queue.cc:1035] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.541594 17025 consensus_queue.cc:1035] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.547551 12587 catalog_manager.cc:5576] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979) does not have the latest schema for tablet c0bfe5dc732047a89e3dc0a6fbe7a120 (table rebalancer_dynamic_table [id=a248e166677c45e39d658bec33c543f7]). Expected version 1 got 0
I20240502 08:18:22.549142 12586 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet c0bfe5dc732047a89e3dc0a6fbe7a120 (table rebalancer_dynamic_table [id=a248e166677c45e39d658bec33c543f7]). Expected version 1 got 0
I20240502 08:18:22.552261 17070 tablet.cc:1722] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:22.552809 17070 tablet.cc:1620] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:22.553738 17072 tablet.cc:1722] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:22.554579 17072 tablet.cc:1620] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:22.556339 12587 catalog_manager.cc:5576] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979) does not have the latest schema for tablet c0bfe5dc732047a89e3dc0a6fbe7a120 (table rebalancer_dynamic_table [id=a248e166677c45e39d658bec33c543f7]). Expected version 1 got 0
I20240502 08:18:22.564628 17073 tablet.cc:1722] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:22.565232 17073 tablet.cc:1620] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:22.567234 12584 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet c0bfe5dc732047a89e3dc0a6fbe7a120 (table rebalancer_dynamic_table [id=a248e166677c45e39d658bec33c543f7]). Expected version 1 got 0
I20240502 08:18:22.569739 16865 consensus_queue.cc:1035] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.570420 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=a248e166677c45e39d658bec33c543f7] alter complete (version 1)
I20240502 08:18:22.580354 16865 consensus_queue.cc:1035] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.602579 16865 consensus_queue.cc:1035] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.607720 17054 consensus_queue.cc:1035] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.677372 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:22.684958 17070 tablet.cc:1722] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:22.685416 17070 tablet.cc:1620] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:22.685680 17073 tablet.cc:1722] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:22.686121 17073 tablet.cc:1620] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:22.686228 17072 tablet.cc:1722] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:22.686709 17072 tablet.cc:1620] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:22.694559 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=a248e166677c45e39d658bec33c543f7] alter complete (version 2)
I20240502 08:18:22.797536 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:22.806155 17070 tablet.cc:1722] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:22.806540 17072 tablet.cc:1722] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:22.807279 17073 tablet.cc:1722] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:22.817008 12587 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=a248e166677c45e39d658bec33c543f7] alter complete (version 3)
I20240502 08:18:22.920099 12586 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:22.920328 12586 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:22.923117 12586 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet c0bfe5dc732047a89e3dc0a6fbe7a120
I20240502 08:18:22.923789 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet c0bfe5dc732047a89e3dc0a6fbe7a120 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:22 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:22.923979 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet c0bfe5dc732047a89e3dc0a6fbe7a120 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:22 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:22.924000 15734 tablet_service.cc:1514] Processing DeleteTablet for tablet c0bfe5dc732047a89e3dc0a6fbe7a120 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:22 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:22.924384 17082 tablet_replica.cc:330] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:22.924681 17082 raft_consensus.cc:2229] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:22.924794 17084 tablet_replica.cc:330] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:22.924959 17083 tablet_replica.cc:330] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:22.925002 17084 raft_consensus.cc:2229] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:22.925114 17082 raft_consensus.cc:2258] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:22.925161 17083 raft_consensus.cc:2229] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:22.925295 17084 raft_consensus.cc:2258] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:22.925416 17083 raft_consensus.cc:2258] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:22.925971 17082 ts_tablet_manager.cc:1875] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:22.926117 17084 ts_tablet_manager.cc:1875] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:22.926259 17083 ts_tablet_manager.cc:1875] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:22.931735 17082 ts_tablet_manager.cc:1888] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:22.931912 17082 log.cc:1196] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/c0bfe5dc732047a89e3dc0a6fbe7a120
I20240502 08:18:22.932320 17082 ts_tablet_manager.cc:1909] T c0bfe5dc732047a89e3dc0a6fbe7a120 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:22.932353 17084 ts_tablet_manager.cc:1888] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:22.932499 17084 log.cc:1196] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/c0bfe5dc732047a89e3dc0a6fbe7a120
I20240502 08:18:22.932569 17083 ts_tablet_manager.cc:1888] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:22.932679 17083 log.cc:1196] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/c0bfe5dc732047a89e3dc0a6fbe7a120
I20240502 08:18:22.933075 17084 ts_tablet_manager.cc:1909] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:22.933179 17083 ts_tablet_manager.cc:1909] T c0bfe5dc732047a89e3dc0a6fbe7a120 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:22.933207 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet c0bfe5dc732047a89e3dc0a6fbe7a120 (table rebalancer_dynamic_table_new_name [id=a248e166677c45e39d658bec33c543f7]) successfully deleted
I20240502 08:18:22.934247 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet c0bfe5dc732047a89e3dc0a6fbe7a120 (table rebalancer_dynamic_table_new_name [id=a248e166677c45e39d658bec33c543f7]) successfully deleted
I20240502 08:18:22.934511 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet c0bfe5dc732047a89e3dc0a6fbe7a120 (table rebalancer_dynamic_table_new_name [id=a248e166677c45e39d658bec33c543f7]) successfully deleted
I20240502 08:18:22.939546 16854 consensus_queue.cc:1035] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.944526 16854 consensus_queue.cc:1035] T 8d5a4eb8278c45f1a8389397a8748070 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:22.971323 12586 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:22.980717 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2cda653a044a41bbf61dff7bb29fe2 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=52eb1ee601914170a78e4182cb32211c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.981149 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f2cda653a044a41bbf61dff7bb29fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.981437 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2cda653a044a41bbf61dff7bb29fe2 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=52eb1ee601914170a78e4182cb32211c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.981493 15468 tablet_service.cc:1467] Processing CreateTablet for tablet b45f9d8096144753a6666e34c457b293 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=52eb1ee601914170a78e4182cb32211c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.981815 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f2cda653a044a41bbf61dff7bb29fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.981904 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b45f9d8096144753a6666e34c457b293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.984776 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2cda653a044a41bbf61dff7bb29fe2 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=52eb1ee601914170a78e4182cb32211c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.985189 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f2cda653a044a41bbf61dff7bb29fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.985404 15734 tablet_service.cc:1467] Processing CreateTablet for tablet b45f9d8096144753a6666e34c457b293 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=52eb1ee601914170a78e4182cb32211c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.985822 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b45f9d8096144753a6666e34c457b293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.990561 17091 tablet_bootstrap.cc:492] T b45f9d8096144753a6666e34c457b293 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:22.992168 15866 tablet_service.cc:1467] Processing CreateTablet for tablet b45f9d8096144753a6666e34c457b293 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=52eb1ee601914170a78e4182cb32211c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.992578 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b45f9d8096144753a6666e34c457b293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.993687 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9d9100dbaa4861ba1d2a1455f9756d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=52eb1ee601914170a78e4182cb32211c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.994037 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9d9100dbaa4861ba1d2a1455f9756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.995144 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9d9100dbaa4861ba1d2a1455f9756d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=52eb1ee601914170a78e4182cb32211c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.995487 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9d9100dbaa4861ba1d2a1455f9756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:22.995594 12710 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9d9100dbaa4861ba1d2a1455f9756d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=52eb1ee601914170a78e4182cb32211c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:22.995954 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9d9100dbaa4861ba1d2a1455f9756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.000461 17091 tablet_bootstrap.cc:654] T b45f9d8096144753a6666e34c457b293 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.009146 17094 tablet_bootstrap.cc:492] T 8f2cda653a044a41bbf61dff7bb29fe2 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:23.010110 17096 tablet_bootstrap.cc:492] T b45f9d8096144753a6666e34c457b293 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:23.011395 17091 tablet_bootstrap.cc:492] T b45f9d8096144753a6666e34c457b293 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:23.011668 17094 tablet_bootstrap.cc:654] T 8f2cda653a044a41bbf61dff7bb29fe2 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.011724 17091 ts_tablet_manager.cc:1367] T b45f9d8096144753a6666e34c457b293 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.005s
I20240502 08:18:23.013007 17092 tablet_bootstrap.cc:492] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:23.013675 17096 tablet_bootstrap.cc:654] T b45f9d8096144753a6666e34c457b293 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.014595 17092 tablet_bootstrap.cc:654] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.014627 17091 raft_consensus.cc:348] T b45f9d8096144753a6666e34c457b293 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.017066 17091 raft_consensus.cc:374] T b45f9d8096144753a6666e34c457b293 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.017444 17091 raft_consensus.cc:729] T b45f9d8096144753a6666e34c457b293 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.017913 17091 consensus_queue.cc:260] T b45f9d8096144753a6666e34c457b293 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.020957 17091 ts_tablet_manager.cc:1398] T b45f9d8096144753a6666e34c457b293 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 08:18:23.022157 17091 tablet_bootstrap.cc:492] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:23.022269 17093 tablet_bootstrap.cc:492] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:23.027187 17091 tablet_bootstrap.cc:654] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.027585 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:23.028077 17093 tablet_bootstrap.cc:654] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.031016 17090 tablet_bootstrap.cc:492] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:23.032181 17094 tablet_bootstrap.cc:492] T 8f2cda653a044a41bbf61dff7bb29fe2 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:23.032302 17094 ts_tablet_manager.cc:1367] T 8f2cda653a044a41bbf61dff7bb29fe2 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.012s
I20240502 08:18:23.032584 17090 tablet_bootstrap.cc:654] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.033143 17094 raft_consensus.cc:348] T 8f2cda653a044a41bbf61dff7bb29fe2 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:23.033341 17094 raft_consensus.cc:374] T 8f2cda653a044a41bbf61dff7bb29fe2 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.033450 17094 raft_consensus.cc:729] T 8f2cda653a044a41bbf61dff7bb29fe2 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.033692 17094 consensus_queue.cc:260] T 8f2cda653a044a41bbf61dff7bb29fe2 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:23.034258 17094 ts_tablet_manager.cc:1398] T 8f2cda653a044a41bbf61dff7bb29fe2 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:23.034636 17094 tablet_bootstrap.cc:492] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:23.036123 17094 tablet_bootstrap.cc:654] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.041255 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 58ba9350fdba4dcdb0754e1263bba3f0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c681b9185aef4df2b4d0dc2fc74e5328]), partition=
I20240502 08:18:23.041572 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58ba9350fdba4dcdb0754e1263bba3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.041734 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 58ba9350fdba4dcdb0754e1263bba3f0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c681b9185aef4df2b4d0dc2fc74e5328]), partition=
I20240502 08:18:23.042105 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58ba9350fdba4dcdb0754e1263bba3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.048717 15866 tablet_service.cc:1467] Processing CreateTablet for tablet 58ba9350fdba4dcdb0754e1263bba3f0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c681b9185aef4df2b4d0dc2fc74e5328]), partition=
I20240502 08:18:23.049053 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58ba9350fdba4dcdb0754e1263bba3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.058697 17091 tablet_bootstrap.cc:492] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:23.058908 17091 ts_tablet_manager.cc:1367] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.005s
I20240502 08:18:23.060366 17091 raft_consensus.cc:348] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.060839 17091 raft_consensus.cc:374] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.061069 17091 raft_consensus.cc:729] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.061407 17091 consensus_queue.cc:260] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.062167 17091 ts_tablet_manager.cc:1398] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:23.062561 17091 tablet_bootstrap.cc:492] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:23.063555 17093 tablet_bootstrap.cc:492] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:23.063746 17093 ts_tablet_manager.cc:1367] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.005s
I20240502 08:18:23.064606 17093 raft_consensus.cc:348] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.064867 17093 raft_consensus.cc:374] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.064993 17093 raft_consensus.cc:729] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.065359 17093 consensus_queue.cc:260] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.067045 17093 ts_tablet_manager.cc:1398] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:18:23.067380 17093 tablet_bootstrap.cc:492] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:23.069994 17091 tablet_bootstrap.cc:654] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.070433 17093 tablet_bootstrap.cc:654] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.075346 17093 tablet_bootstrap.cc:492] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:23.075557 17093 ts_tablet_manager.cc:1367] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240502 08:18:23.076308 17093 raft_consensus.cc:348] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:23.076579 17093 raft_consensus.cc:374] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.076694 17093 raft_consensus.cc:729] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.076975 17093 consensus_queue.cc:260] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:23.078862 17093 ts_tablet_manager.cc:1398] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:23.079916 17091 tablet_bootstrap.cc:492] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:23.080114 17091 ts_tablet_manager.cc:1367] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 08:18:23.080919 17091 raft_consensus.cc:348] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:23.081178 17091 raft_consensus.cc:374] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.081303 17091 raft_consensus.cc:729] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.081568 17091 consensus_queue.cc:260] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:23.084870 17091 ts_tablet_manager.cc:1398] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:18:23.111409 17096 tablet_bootstrap.cc:492] T b45f9d8096144753a6666e34c457b293 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:23.111641 17096 ts_tablet_manager.cc:1367] T b45f9d8096144753a6666e34c457b293 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.102s user 0.006s sys 0.003s
I20240502 08:18:23.112468 17096 raft_consensus.cc:348] T b45f9d8096144753a6666e34c457b293 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.112767 17096 raft_consensus.cc:374] T b45f9d8096144753a6666e34c457b293 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.114405 17096 raft_consensus.cc:729] T b45f9d8096144753a6666e34c457b293 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.114714 17096 consensus_queue.cc:260] T b45f9d8096144753a6666e34c457b293 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.115939 17110 tablet_bootstrap.cc:492] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:23.117009 17096 ts_tablet_manager.cc:1398] T b45f9d8096144753a6666e34c457b293 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:23.119560 17090 tablet_bootstrap.cc:492] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:23.119724 17110 tablet_bootstrap.cc:654] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.119926 17090 ts_tablet_manager.cc:1367] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.089s user 0.005s sys 0.021s
I20240502 08:18:23.120754 17094 tablet_bootstrap.cc:492] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:23.120994 17094 ts_tablet_manager.cc:1367] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.086s user 0.029s sys 0.037s
I20240502 08:18:23.121479 17092 tablet_bootstrap.cc:492] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:23.122318 17090 raft_consensus.cc:348] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:23.122344 17094 raft_consensus.cc:348] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.122555 17090 raft_consensus.cc:374] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.122581 17094 raft_consensus.cc:374] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.122648 17090 raft_consensus.cc:729] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.122669 17094 raft_consensus.cc:729] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.122849 17090 consensus_queue.cc:260] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:23.124502 17094 consensus_queue.cc:260] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.126849 17090 ts_tablet_manager.cc:1398] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:23.128369 17092 ts_tablet_manager.cc:1367] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.021s
I20240502 08:18:23.128590 17094 ts_tablet_manager.cc:1398] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:23.129216 17092 raft_consensus.cc:348] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.129622 17092 raft_consensus.cc:374] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.130892 17092 raft_consensus.cc:729] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.131184 17092 consensus_queue.cc:260] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.131841 17092 ts_tablet_manager.cc:1398] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:18:23.132197 17092 tablet_bootstrap.cc:492] T 8f2cda653a044a41bbf61dff7bb29fe2 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:23.134905 17092 tablet_bootstrap.cc:654] T 8f2cda653a044a41bbf61dff7bb29fe2 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.137900 17092 tablet_bootstrap.cc:492] T 8f2cda653a044a41bbf61dff7bb29fe2 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:23.138075 17092 ts_tablet_manager.cc:1367] T 8f2cda653a044a41bbf61dff7bb29fe2 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 08:18:23.138813 17092 raft_consensus.cc:348] T 8f2cda653a044a41bbf61dff7bb29fe2 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:23.139019 17092 raft_consensus.cc:374] T 8f2cda653a044a41bbf61dff7bb29fe2 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.139101 17092 raft_consensus.cc:729] T 8f2cda653a044a41bbf61dff7bb29fe2 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.139343 17110 tablet_bootstrap.cc:492] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:23.139322 17092 consensus_queue.cc:260] T 8f2cda653a044a41bbf61dff7bb29fe2 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:23.139479 17110 ts_tablet_manager.cc:1367] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.004s
I20240502 08:18:23.139833 17092 ts_tablet_manager.cc:1398] T 8f2cda653a044a41bbf61dff7bb29fe2 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:23.140167 17110 raft_consensus.cc:348] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:23.140542 17110 raft_consensus.cc:374] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.140645 17110 raft_consensus.cc:729] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.140859 17110 consensus_queue.cc:260] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:23.141479 17110 ts_tablet_manager.cc:1398] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:23.141467 17087 raft_consensus.cc:482] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:23.141610 17087 raft_consensus.cc:504] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.142299 17087 leader_election.cc:288] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:23.143028 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d9100dbaa4861ba1d2a1455f9756d" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:23.143026 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d9100dbaa4861ba1d2a1455f9756d" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:23.143273 12727 raft_consensus.cc:2454] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:23.143276 15620 raft_consensus.cc:2454] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:23.143641 15690 leader_election.cc:302] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:23.143910 17087 raft_consensus.cc:2781] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:23.144027 17087 raft_consensus.cc:482] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:23.144137 17087 raft_consensus.cc:3037] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.146502 17087 raft_consensus.cc:504] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.147051 17087 leader_election.cc:288] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:23.147397 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d9100dbaa4861ba1d2a1455f9756d" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:23.147527 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9d9100dbaa4861ba1d2a1455f9756d" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:23.147578 15620 raft_consensus.cc:3037] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.147716 12727 raft_consensus.cc:3037] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.149425 12727 raft_consensus.cc:2454] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:23.149752 15688 leader_election.cc:302] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:23.150017 17087 raft_consensus.cc:2781] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:23.150017 15620 raft_consensus.cc:2454] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:23.150130 17087 raft_consensus.cc:686] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:23.150669 17087 consensus_queue.cc:237] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.154175 12579 catalog_manager.cc:5484] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:23.157416 16632 raft_consensus.cc:482] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:23.157601 16632 raft_consensus.cc:504] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.158196 16632 leader_election.cc:288] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:23.158617 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45f9d8096144753a6666e34c457b293" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:23.158803 15487 raft_consensus.cc:2454] T b45f9d8096144753a6666e34c457b293 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:23.158815 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45f9d8096144753a6666e34c457b293" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:23.158980 15754 raft_consensus.cc:2454] T b45f9d8096144753a6666e34c457b293 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:23.159304 15819 leader_election.cc:302] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:23.159587 16632 raft_consensus.cc:2781] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:23.159677 16632 raft_consensus.cc:482] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:23.159752 16632 raft_consensus.cc:3037] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.161414 16632 raft_consensus.cc:504] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.161991 16632 leader_election.cc:288] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:23.162276 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45f9d8096144753a6666e34c457b293" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:23.162513 15487 raft_consensus.cc:3037] T b45f9d8096144753a6666e34c457b293 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.164523 15487 raft_consensus.cc:2454] T b45f9d8096144753a6666e34c457b293 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:23.164848 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45f9d8096144753a6666e34c457b293" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:23.164889 15819 leader_election.cc:302] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:23.165027 15754 raft_consensus.cc:3037] T b45f9d8096144753a6666e34c457b293 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.165148 16632 raft_consensus.cc:2781] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:23.165313 16632 raft_consensus.cc:686] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:23.165555 16632 consensus_queue.cc:237] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.167419 15754 raft_consensus.cc:2454] T b45f9d8096144753a6666e34c457b293 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:23.169153 12579 catalog_manager.cc:5484] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:23.216552 17105 raft_consensus.cc:482] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:23.216749 17105 raft_consensus.cc:504] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:23.217543 17105 leader_election.cc:288] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:23.217849 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2cda653a044a41bbf61dff7bb29fe2" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:23.218151 15620 raft_consensus.cc:2454] T 8f2cda653a044a41bbf61dff7bb29fe2 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:23.218124 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2cda653a044a41bbf61dff7bb29fe2" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:23.218329 12727 raft_consensus.cc:2454] T 8f2cda653a044a41bbf61dff7bb29fe2 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:23.218582 12928 leader_election.cc:302] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:23.218852 17105 raft_consensus.cc:2781] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:23.218979 17105 raft_consensus.cc:482] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:23.219062 17105 raft_consensus.cc:3037] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.221144 17105 raft_consensus.cc:504] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:23.221632 17105 leader_election.cc:288] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:23.222098 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2cda653a044a41bbf61dff7bb29fe2" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:23.222133 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2cda653a044a41bbf61dff7bb29fe2" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:23.222272 15621 raft_consensus.cc:3037] T 8f2cda653a044a41bbf61dff7bb29fe2 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.222287 12727 raft_consensus.cc:3037] T 8f2cda653a044a41bbf61dff7bb29fe2 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.224113 12727 raft_consensus.cc:2454] T 8f2cda653a044a41bbf61dff7bb29fe2 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:23.224442 15621 raft_consensus.cc:2454] T 8f2cda653a044a41bbf61dff7bb29fe2 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:23.224458 12926 leader_election.cc:302] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:18:23.224725 17105 raft_consensus.cc:2781] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:23.224889 17105 raft_consensus.cc:686] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:23.225165 17105 consensus_queue.cc:237] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:23.228724 12579 catalog_manager.cc:5484] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:23.242635 16632 raft_consensus.cc:482] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:23.242771 16632 raft_consensus.cc:504] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:23.243903 16632 leader_election.cc:288] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:23.244030 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58ba9350fdba4dcdb0754e1263bba3f0" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:23.244237 15755 raft_consensus.cc:2454] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:23.244593 15818 leader_election.cc:302] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:23.244594 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58ba9350fdba4dcdb0754e1263bba3f0" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:23.244832 16632 raft_consensus.cc:2781] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:23.244813 12861 raft_consensus.cc:2454] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:23.244939 16632 raft_consensus.cc:482] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:23.245047 16632 raft_consensus.cc:3037] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.247251 16632 raft_consensus.cc:504] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:23.247905 16632 leader_election.cc:288] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:23.248337 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58ba9350fdba4dcdb0754e1263bba3f0" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:23.248514 12861 raft_consensus.cc:3037] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.250502 12861 raft_consensus.cc:2454] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:23.250676 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58ba9350fdba4dcdb0754e1263bba3f0" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:23.250830 15819 leader_election.cc:302] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:23.250867 15755 raft_consensus.cc:3037] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.251083 16632 raft_consensus.cc:2781] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:23.251232 16632 raft_consensus.cc:686] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:23.251494 16632 consensus_queue.cc:237] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:23.253170 15755 raft_consensus.cc:2454] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:23.254846 12579 catalog_manager.cc:5484] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:23.364806 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:23.370102 12859 raft_consensus.cc:1261] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:23.370294 15754 raft_consensus.cc:1261] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:23.370636 16632 consensus_queue.cc:1035] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:23.370896 16865 consensus_queue.cc:1035] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:23.378489 12584 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet 58ba9350fdba4dcdb0754e1263bba3f0 (table rebalancer_dynamic_table [id=c681b9185aef4df2b4d0dc2fc74e5328]). Expected version 1 got 0
I20240502 08:18:23.380193 17121 tablet.cc:1722] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:23.381012 17121 tablet.cc:1620] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:23.384150 12584 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet 58ba9350fdba4dcdb0754e1263bba3f0 (table rebalancer_dynamic_table [id=c681b9185aef4df2b4d0dc2fc74e5328]). Expected version 1 got 0
I20240502 08:18:23.392551 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=c681b9185aef4df2b4d0dc2fc74e5328] alter complete (version 1)
I20240502 08:18:23.393792 17123 tablet.cc:1722] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:23.394331 17123 tablet.cc:1620] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:23.395190 17122 tablet.cc:1722] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:23.398699 12584 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet 58ba9350fdba4dcdb0754e1263bba3f0 (table rebalancer_dynamic_table [id=c681b9185aef4df2b4d0dc2fc74e5328]). Expected version 1 got 0
I20240502 08:18:23.399389 12579 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet 58ba9350fdba4dcdb0754e1263bba3f0 (table rebalancer_dynamic_table [id=c681b9185aef4df2b4d0dc2fc74e5328]). Expected version 1 got 0
I20240502 08:18:23.400148 17122 tablet.cc:1620] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:23.495690 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:23.502434 17121 tablet.cc:1722] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:23.502981 17121 tablet.cc:1620] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:23.503075 17123 tablet.cc:1722] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:23.503594 17123 tablet.cc:1620] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:23.505112 17122 tablet.cc:1722] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:23.505579 17122 tablet.cc:1620] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:23.512266 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=c681b9185aef4df2b4d0dc2fc74e5328] alter complete (version 2)
I20240502 08:18:23.591573 16854 consensus_queue.cc:1035] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:23.595949 17087 consensus_queue.cc:1035] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:23.614671 12587 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:23.622057 17121 tablet.cc:1722] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:23.623093 17122 tablet.cc:1722] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:23.624017 17123 tablet.cc:1722] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:23.632578 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=c681b9185aef4df2b4d0dc2fc74e5328] alter complete (version 3)
I20240502 08:18:23.706707 17064 consensus_queue.cc:1035] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:23.711294 17064 consensus_queue.cc:1035] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:23.728051 16865 consensus_queue.cc:1035] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:23.731779 16865 consensus_queue.cc:1035] T b45f9d8096144753a6666e34c457b293 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:23.737087 12587 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:23.737339 12587 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:23.741020 12587 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 58ba9350fdba4dcdb0754e1263bba3f0
I20240502 08:18:23.741994 15735 tablet_service.cc:1514] Processing DeleteTablet for tablet 58ba9350fdba4dcdb0754e1263bba3f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:23 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:23.742063 15866 tablet_service.cc:1514] Processing DeleteTablet for tablet 58ba9350fdba4dcdb0754e1263bba3f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:23 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:23.742386 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet 58ba9350fdba4dcdb0754e1263bba3f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:23 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:23.742821 17135 tablet_replica.cc:330] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:23.743044 17135 raft_consensus.cc:2229] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:23.743440 17135 raft_consensus.cc:2258] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:23.743921 17136 tablet_replica.cc:330] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:23.744342 17135 ts_tablet_manager.cc:1875] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:23.744619 17136 raft_consensus.cc:2229] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:23.744820 17136 raft_consensus.cc:2258] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:23.745867 17136 ts_tablet_manager.cc:1875] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:23.751569 17134 tablet_replica.cc:330] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:23.751798 17134 raft_consensus.cc:2229] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:23.752049 17134 raft_consensus.cc:2258] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:23.752795 17135 ts_tablet_manager.cc:1888] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:23.752916 17135 log.cc:1196] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/58ba9350fdba4dcdb0754e1263bba3f0
I20240502 08:18:23.753404 17135 ts_tablet_manager.cc:1909] T 58ba9350fdba4dcdb0754e1263bba3f0 P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:23.753969 17134 ts_tablet_manager.cc:1875] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:23.754323 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 58ba9350fdba4dcdb0754e1263bba3f0 (table rebalancer_dynamic_table_new_name [id=c681b9185aef4df2b4d0dc2fc74e5328]) successfully deleted
I20240502 08:18:23.759974 17136 ts_tablet_manager.cc:1888] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:23.760146 17136 log.cc:1196] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/58ba9350fdba4dcdb0754e1263bba3f0
I20240502 08:18:23.760653 17136 ts_tablet_manager.cc:1909] T 58ba9350fdba4dcdb0754e1263bba3f0 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:23.761758 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 58ba9350fdba4dcdb0754e1263bba3f0 (table rebalancer_dynamic_table_new_name [id=c681b9185aef4df2b4d0dc2fc74e5328]) successfully deleted
I20240502 08:18:23.762249 17134 ts_tablet_manager.cc:1888] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:23.762379 17134 log.cc:1196] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/58ba9350fdba4dcdb0754e1263bba3f0
I20240502 08:18:23.762769 17134 ts_tablet_manager.cc:1909] T 58ba9350fdba4dcdb0754e1263bba3f0 P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:23.763628 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 58ba9350fdba4dcdb0754e1263bba3f0 (table rebalancer_dynamic_table_new_name [id=c681b9185aef4df2b4d0dc2fc74e5328]) successfully deleted
I20240502 08:18:23.764865 12587 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:23.775290 12842 tablet_service.cc:1467] Processing CreateTablet for tablet c24538742b874c92a70e17b3450993fb (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=bdfb8acace12439cb1fe7b02ffd810cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:23.775717 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24538742b874c92a70e17b3450993fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.775959 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 3da17119cfad49fb80131d2e46f125c8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=bdfb8acace12439cb1fe7b02ffd810cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:23.776333 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da17119cfad49fb80131d2e46f125c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.776582 15866 tablet_service.cc:1467] Processing CreateTablet for tablet 3da17119cfad49fb80131d2e46f125c8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=bdfb8acace12439cb1fe7b02ffd810cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:23.776989 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da17119cfad49fb80131d2e46f125c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.779682 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 02ec4a224738406185f93fccb3b9320e (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=bdfb8acace12439cb1fe7b02ffd810cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:23.780089 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ec4a224738406185f93fccb3b9320e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.782872 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 02ec4a224738406185f93fccb3b9320e (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=bdfb8acace12439cb1fe7b02ffd810cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:23.782874 12709 tablet_service.cc:1467] Processing CreateTablet for tablet c24538742b874c92a70e17b3450993fb (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=bdfb8acace12439cb1fe7b02ffd810cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:23.783267 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ec4a224738406185f93fccb3b9320e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.783310 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24538742b874c92a70e17b3450993fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.785259 17140 tablet_bootstrap.cc:492] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:23.785820 17139 tablet_bootstrap.cc:492] T 3da17119cfad49fb80131d2e46f125c8 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:23.786875 12710 tablet_service.cc:1467] Processing CreateTablet for tablet 02ec4a224738406185f93fccb3b9320e (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=bdfb8acace12439cb1fe7b02ffd810cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:23.786976 17140 tablet_bootstrap.cc:654] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.787276 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ec4a224738406185f93fccb3b9320e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.787653 17139 tablet_bootstrap.cc:654] T 3da17119cfad49fb80131d2e46f125c8 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.794224 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 3da17119cfad49fb80131d2e46f125c8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=bdfb8acace12439cb1fe7b02ffd810cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:23.794690 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3da17119cfad49fb80131d2e46f125c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.797246 17140 tablet_bootstrap.cc:492] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:23.797365 17143 tablet_bootstrap.cc:492] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:23.797375 17140 ts_tablet_manager.cc:1367] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240502 08:18:23.798162 17140 raft_consensus.cc:348] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.798378 17140 raft_consensus.cc:374] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.798498 17140 raft_consensus.cc:729] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.798734 17140 consensus_queue.cc:260] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.798862 17143 tablet_bootstrap.cc:654] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.804178 17140 ts_tablet_manager.cc:1398] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:18:23.812216 17143 tablet_bootstrap.cc:492] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:23.812484 17143 ts_tablet_manager.cc:1367] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20240502 08:18:23.814132 17143 raft_consensus.cc:348] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.814414 17143 raft_consensus.cc:374] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.814570 17143 raft_consensus.cc:729] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.814939 17143 consensus_queue.cc:260] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.815706 17143 ts_tablet_manager.cc:1398] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:23.823246 17147 tablet_bootstrap.cc:492] T 3da17119cfad49fb80131d2e46f125c8 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:23.824986 17143 tablet_bootstrap.cc:492] T 02ec4a224738406185f93fccb3b9320e P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:23.825714 15866 tablet_service.cc:1467] Processing CreateTablet for tablet c24538742b874c92a70e17b3450993fb (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=bdfb8acace12439cb1fe7b02ffd810cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:23.826150 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24538742b874c92a70e17b3450993fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.827714 17143 tablet_bootstrap.cc:654] T 02ec4a224738406185f93fccb3b9320e P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.829048 17147 tablet_bootstrap.cc:654] T 3da17119cfad49fb80131d2e46f125c8 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.829568 17145 tablet_bootstrap.cc:492] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:23.831265 17145 tablet_bootstrap.cc:654] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.836082 17148 tablet_bootstrap.cc:492] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:23.837813 17143 tablet_bootstrap.cc:492] T 02ec4a224738406185f93fccb3b9320e P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:23.837976 17143 ts_tablet_manager.cc:1367] T 02ec4a224738406185f93fccb3b9320e P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240502 08:18:23.838675 17139 tablet_bootstrap.cc:492] T 3da17119cfad49fb80131d2e46f125c8 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:23.838861 17147 tablet_bootstrap.cc:492] T 3da17119cfad49fb80131d2e46f125c8 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:23.838762 17143 raft_consensus.cc:348] T 02ec4a224738406185f93fccb3b9320e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:23.838874 17139 ts_tablet_manager.cc:1367] T 3da17119cfad49fb80131d2e46f125c8 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.053s user 0.022s sys 0.025s
I20240502 08:18:23.839059 17147 ts_tablet_manager.cc:1367] T 3da17119cfad49fb80131d2e46f125c8 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240502 08:18:23.839743 17139 raft_consensus.cc:348] T 3da17119cfad49fb80131d2e46f125c8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:23.840019 17139 raft_consensus.cc:374] T 3da17119cfad49fb80131d2e46f125c8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.839870 17147 raft_consensus.cc:348] T 3da17119cfad49fb80131d2e46f125c8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:23.840157 17147 raft_consensus.cc:374] T 3da17119cfad49fb80131d2e46f125c8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.840170 17139 raft_consensus.cc:729] T 3da17119cfad49fb80131d2e46f125c8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.840327 17147 raft_consensus.cc:729] T 3da17119cfad49fb80131d2e46f125c8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.840541 17139 consensus_queue.cc:260] T 3da17119cfad49fb80131d2e46f125c8 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:23.840605 17147 consensus_queue.cc:260] T 3da17119cfad49fb80131d2e46f125c8 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:23.838972 17143 raft_consensus.cc:374] T 02ec4a224738406185f93fccb3b9320e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.841058 17143 raft_consensus.cc:729] T 02ec4a224738406185f93fccb3b9320e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.841337 17139 ts_tablet_manager.cc:1398] T 3da17119cfad49fb80131d2e46f125c8 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:23.841331 17117 raft_consensus.cc:482] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:23.841610 17139 tablet_bootstrap.cc:492] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:23.841517 17117 raft_consensus.cc:504] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.842326 17147 ts_tablet_manager.cc:1398] T 3da17119cfad49fb80131d2e46f125c8 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:23.842326 17117 leader_election.cc:288] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:23.843297 17139 tablet_bootstrap.cc:654] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.844161 17148 tablet_bootstrap.cc:654] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.845162 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24538742b874c92a70e17b3450993fb" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:23.845443 12861 raft_consensus.cc:2454] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:23.845440 17143 consensus_queue.cc:260] T 02ec4a224738406185f93fccb3b9320e P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:23.845983 12664 leader_election.cc:302] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:23.846079 17143 ts_tablet_manager.cc:1398] T 02ec4a224738406185f93fccb3b9320e P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240502 08:18:23.846391 16759 raft_consensus.cc:2781] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:23.846501 16759 raft_consensus.cc:482] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:23.846611 16759 raft_consensus.cc:3037] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.846848 12587 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:23.849467 16759 raft_consensus.cc:504] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.850046 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24538742b874c92a70e17b3450993fb" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:23.850528 16759 leader_election.cc:288] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
W20240502 08:18:23.850752 12665 leader_election.cc:341] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:23.851413 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24538742b874c92a70e17b3450993fb" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:23.851621 12861 raft_consensus.cc:3037] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.852571 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24538742b874c92a70e17b3450993fb" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:23.852798 17139 tablet_bootstrap.cc:492] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:23.852991 17139 ts_tablet_manager.cc:1367] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
W20240502 08:18:23.853083 12665 leader_election.cc:341] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:23.853682 17139 raft_consensus.cc:348] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.853904 17139 raft_consensus.cc:374] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.853981 17139 raft_consensus.cc:729] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.854264 12861 raft_consensus.cc:2454] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:23.854213 17139 consensus_queue.cc:260] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.854676 12664 leader_election.cc:302] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters: 10f0e36203914b85b77e78609ffced98
I20240502 08:18:23.854847 17139 ts_tablet_manager.cc:1398] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:23.855154 17117 raft_consensus.cc:2781] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:23.859916 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1e3b48fdd14c6c859aa31ac990a17d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fbb936619eb54c00aebfd01210f7e41a]), partition=
I20240502 08:18:23.860263 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1e3b48fdd14c6c859aa31ac990a17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.860458 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1e3b48fdd14c6c859aa31ac990a17d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fbb936619eb54c00aebfd01210f7e41a]), partition=
I20240502 08:18:23.860798 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1e3b48fdd14c6c859aa31ac990a17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.865499 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1e3b48fdd14c6c859aa31ac990a17d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fbb936619eb54c00aebfd01210f7e41a]), partition=
I20240502 08:18:23.865823 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1e3b48fdd14c6c859aa31ac990a17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:23.871279 17117 raft_consensus.cc:686] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:18:23.871569 17117 consensus_queue.cc:237] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.873428 17140 tablet_bootstrap.cc:492] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:23.874884 17140 tablet_bootstrap.cc:654] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.875919 17146 tablet_bootstrap.cc:492] T 02ec4a224738406185f93fccb3b9320e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:23.878388 12579 catalog_manager.cc:5484] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:23.887427 17145 tablet_bootstrap.cc:492] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:23.887601 17145 ts_tablet_manager.cc:1367] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.003s
I20240502 08:18:23.887679 17140 tablet_bootstrap.cc:492] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:23.887840 17140 ts_tablet_manager.cc:1367] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20240502 08:18:23.888687 17140 raft_consensus.cc:348] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.888917 17140 raft_consensus.cc:374] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.889271 17140 raft_consensus.cc:729] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.889572 17140 consensus_queue.cc:260] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.890072 17140 ts_tablet_manager.cc:1398] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:23.891763 17155 tablet_bootstrap.cc:492] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:23.892925 17148 tablet_bootstrap.cc:492] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:23.893118 17148 ts_tablet_manager.cc:1367] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20240502 08:18:23.893456 17155 tablet_bootstrap.cc:654] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.893739 17146 tablet_bootstrap.cc:654] T 02ec4a224738406185f93fccb3b9320e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.894017 17148 raft_consensus.cc:348] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:23.894287 17148 raft_consensus.cc:374] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.894441 17148 raft_consensus.cc:729] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.894726 17148 consensus_queue.cc:260] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:23.895006 17145 raft_consensus.cc:348] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:23.895232 17145 raft_consensus.cc:374] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.895314 17145 raft_consensus.cc:729] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.895526 17145 consensus_queue.cc:260] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:23.896057 17145 ts_tablet_manager.cc:1398] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 08:18:23.897694 17064 raft_consensus.cc:482] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:23.897826 17064 raft_consensus.cc:504] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:23.898639 17064 leader_election.cc:288] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:23.898939 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da17119cfad49fb80131d2e46f125c8" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:23.899087 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da17119cfad49fb80131d2e46f125c8" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:23.899175 15887 raft_consensus.cc:2454] T 3da17119cfad49fb80131d2e46f125c8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:23.899336 13121 raft_consensus.cc:2454] T 3da17119cfad49fb80131d2e46f125c8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:23.899593 12927 leader_election.cc:302] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:18:23.900033 17064 raft_consensus.cc:2781] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:23.900142 17064 raft_consensus.cc:482] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:23.900216 17064 raft_consensus.cc:3037] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.902482 17148 ts_tablet_manager.cc:1398] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:18:23.902786 17148 tablet_bootstrap.cc:492] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:23.903286 17064 raft_consensus.cc:504] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:23.904191 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da17119cfad49fb80131d2e46f125c8" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:23.904366 17148 tablet_bootstrap.cc:654] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:23.904392 15887 raft_consensus.cc:3037] T 3da17119cfad49fb80131d2e46f125c8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.905900 17146 tablet_bootstrap.cc:492] T 02ec4a224738406185f93fccb3b9320e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:23.906088 17146 ts_tablet_manager.cc:1367] T 02ec4a224738406185f93fccb3b9320e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20240502 08:18:23.906865 17146 raft_consensus.cc:348] T 02ec4a224738406185f93fccb3b9320e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:23.907042 15887 raft_consensus.cc:2454] T 3da17119cfad49fb80131d2e46f125c8 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:23.907109 17146 raft_consensus.cc:374] T 02ec4a224738406185f93fccb3b9320e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.907267 17146 raft_consensus.cc:729] T 02ec4a224738406185f93fccb3b9320e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.907588 17146 consensus_queue.cc:260] T 02ec4a224738406185f93fccb3b9320e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:23.907915 12927 leader_election.cc:302] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:18:23.908219 17105 raft_consensus.cc:2781] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:23.909253 17146 ts_tablet_manager.cc:1398] T 02ec4a224738406185f93fccb3b9320e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:23.910128 17105 raft_consensus.cc:686] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:23.910375 17105 consensus_queue.cc:237] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:23.911062 17155 tablet_bootstrap.cc:492] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:23.911245 17155 ts_tablet_manager.cc:1367] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.000s
I20240502 08:18:23.911295 17148 tablet_bootstrap.cc:492] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:23.911636 17064 leader_election.cc:288] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:23.912245 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3da17119cfad49fb80131d2e46f125c8" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:23.912475 13121 raft_consensus.cc:3037] T 3da17119cfad49fb80131d2e46f125c8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.912618 17148 ts_tablet_manager.cc:1367] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20240502 08:18:23.913373 17148 raft_consensus.cc:348] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.913578 17148 raft_consensus.cc:374] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.913651 17148 raft_consensus.cc:729] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.913897 17148 consensus_queue.cc:260] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.915503 13121 raft_consensus.cc:2454] T 3da17119cfad49fb80131d2e46f125c8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:23.915987 17148 ts_tablet_manager.cc:1398] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:23.918263 17155 raft_consensus.cc:348] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.918553 17155 raft_consensus.cc:374] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:23.918685 17155 raft_consensus.cc:729] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:23.918944 17155 consensus_queue.cc:260] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.919471 17155 ts_tablet_manager.cc:1398] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:18:23.919745 12584 catalog_manager.cc:5484] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:23.927090 17081 raft_consensus.cc:482] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:23.927225 17081 raft_consensus.cc:504] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:23.927994 17081 leader_election.cc:288] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:23.928855 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ec4a224738406185f93fccb3b9320e" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:23.929085 12727 raft_consensus.cc:2454] T 02ec4a224738406185f93fccb3b9320e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:23.929368 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ec4a224738406185f93fccb3b9320e" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:23.929575 15621 raft_consensus.cc:2454] T 02ec4a224738406185f93fccb3b9320e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:23.929867 15419 leader_election.cc:302] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:23.930495 17000 raft_consensus.cc:2781] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:23.930588 17000 raft_consensus.cc:482] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:23.930703 17000 raft_consensus.cc:3037] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.932718 17000 raft_consensus.cc:504] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:23.933269 17000 leader_election.cc:288] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:23.933552 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ec4a224738406185f93fccb3b9320e" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:23.933739 12727 raft_consensus.cc:3037] T 02ec4a224738406185f93fccb3b9320e P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.933763 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ec4a224738406185f93fccb3b9320e" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:23.933941 15621 raft_consensus.cc:3037] T 02ec4a224738406185f93fccb3b9320e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.936058 12727 raft_consensus.cc:2454] T 02ec4a224738406185f93fccb3b9320e P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:23.936110 15621 raft_consensus.cc:2454] T 02ec4a224738406185f93fccb3b9320e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:23.936419 15419 leader_election.cc:302] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:23.936683 17000 raft_consensus.cc:2781] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:23.936830 17000 raft_consensus.cc:686] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:23.937059 17000 consensus_queue.cc:237] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:23.940109 12579 catalog_manager.cc:5484] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:23.963948 16865 raft_consensus.cc:482] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:23.964105 16865 raft_consensus.cc:504] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:23.964708 16865 leader_election.cc:288] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:23.965116 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24538742b874c92a70e17b3450993fb" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:23.965274 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24538742b874c92a70e17b3450993fb" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:23.965523 12861 raft_consensus.cc:2379] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10f0e36203914b85b77e78609ffced98 in current term 1: Already voted for candidate 29b035434087435cbaf4ebcac98eedee in this term.
I20240502 08:18:23.965935 15819 leader_election.cc:302] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10f0e36203914b85b77e78609ffced98; no voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a
I20240502 08:18:23.966234 16632 raft_consensus.cc:3037] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:23.968261 16632 raft_consensus.cc:2726] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:23.998977 17025 raft_consensus.cc:482] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:23.999146 17025 raft_consensus.cc:504] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:23.999855 17025 leader_election.cc:288] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:24.000191 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1e3b48fdd14c6c859aa31ac990a17d" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:24.000269 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1e3b48fdd14c6c859aa31ac990a17d" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:24.000401 15487 raft_consensus.cc:2454] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:24.000530 15755 raft_consensus.cc:2454] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:24.000880 12795 leader_election.cc:302] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:24.001195 17025 raft_consensus.cc:2781] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:24.001308 17025 raft_consensus.cc:482] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:24.001399 17025 raft_consensus.cc:3037] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.003582 17025 raft_consensus.cc:504] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:24.004195 17025 leader_election.cc:288] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:24.004648 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1e3b48fdd14c6c859aa31ac990a17d" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:24.004715 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1e3b48fdd14c6c859aa31ac990a17d" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:24.004824 15487 raft_consensus.cc:3037] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.004917 15755 raft_consensus.cc:3037] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.007030 15755 raft_consensus.cc:2454] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:24.007050 15487 raft_consensus.cc:2454] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:24.007431 12795 leader_election.cc:302] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:24.007799 17025 raft_consensus.cc:2781] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:24.008176 17025 raft_consensus.cc:686] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:18:24.008515 17025 consensus_queue.cc:237] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:24.011926 12579 catalog_manager.cc:5484] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:24.128629 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:24.134366 15755 raft_consensus.cc:1261] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:24.134912 15487 raft_consensus.cc:1261] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:24.134876 17025 consensus_queue.cc:1035] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:24.135438 17025 consensus_queue.cc:1035] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:24.139102 12579 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet 9f1e3b48fdd14c6c859aa31ac990a17d (table rebalancer_dynamic_table [id=fbb936619eb54c00aebfd01210f7e41a]). Expected version 1 got 0
I20240502 08:18:24.139456 17171 tablet.cc:1722] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:24.140044 17171 tablet.cc:1620] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:24.140260 17170 tablet.cc:1722] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:24.140514 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet 9f1e3b48fdd14c6c859aa31ac990a17d (table rebalancer_dynamic_table [id=fbb936619eb54c00aebfd01210f7e41a]). Expected version 1 got 0
I20240502 08:18:24.140677 17170 tablet.cc:1620] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:24.145539 17172 tablet.cc:1722] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:24.146164 17172 tablet.cc:1620] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:24.148624 12579 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet 9f1e3b48fdd14c6c859aa31ac990a17d (table rebalancer_dynamic_table [id=fbb936619eb54c00aebfd01210f7e41a]). Expected version 1 got 0
I20240502 08:18:24.150624 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=fbb936619eb54c00aebfd01210f7e41a] alter complete (version 1)
I20240502 08:18:24.253123 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:24.260143 17170 tablet.cc:1722] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:24.260763 17170 tablet.cc:1620] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:24.261853 17171 tablet.cc:1722] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:24.262399 17171 tablet.cc:1620] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:24.274448 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=fbb936619eb54c00aebfd01210f7e41a] alter complete (version 2)
I20240502 08:18:24.283249 17172 tablet.cc:1722] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:24.283685 17172 tablet.cc:1620] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:24.321203 17153 consensus_queue.cc:1035] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:24.334757 17153 consensus_queue.cc:1035] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:24.378521 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:24.379930 17081 consensus_queue.cc:1035] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:24.384567 17000 consensus_queue.cc:1035] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:24.387992 17170 tablet.cc:1722] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:24.388597 17172 tablet.cc:1722] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:24.389763 17171 tablet.cc:1722] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:24.402523 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=fbb936619eb54c00aebfd01210f7e41a] alter complete (version 3)
I20240502 08:18:24.406725 17064 consensus_queue.cc:1035] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:24.410847 17064 consensus_queue.cc:1035] T 3da17119cfad49fb80131d2e46f125c8 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:24.447942 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:24.457362 13101 tablet_service.cc:1467] Processing CreateTablet for tablet fd0a5745bdba436e848ba33029ef01b4 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2f80953a600947b79f5e500d1cb36f09]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:24.457432 15599 tablet_service.cc:1467] Processing CreateTablet for tablet fd0a5745bdba436e848ba33029ef01b4 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2f80953a600947b79f5e500d1cb36f09]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:24.457780 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0a5745bdba436e848ba33029ef01b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.457791 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0a5745bdba436e848ba33029ef01b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.459228 15866 tablet_service.cc:1467] Processing CreateTablet for tablet 6faeb5cba6044a46907cc82448e43234 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2f80953a600947b79f5e500d1cb36f09]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:24.459690 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6faeb5cba6044a46907cc82448e43234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.459831 15735 tablet_service.cc:1467] Processing CreateTablet for tablet fd0a5745bdba436e848ba33029ef01b4 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2f80953a600947b79f5e500d1cb36f09]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:24.460215 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0a5745bdba436e848ba33029ef01b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.462769 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 6faeb5cba6044a46907cc82448e43234 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2f80953a600947b79f5e500d1cb36f09]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:24.463121 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6faeb5cba6044a46907cc82448e43234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.466773 15733 tablet_service.cc:1467] Processing CreateTablet for tablet f50084f1f7784cbfbcb5cb338dbd192e (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2f80953a600947b79f5e500d1cb36f09]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:24.467111 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50084f1f7784cbfbcb5cb338dbd192e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.473963 17185 tablet_bootstrap.cc:492] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:24.474921 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 6faeb5cba6044a46907cc82448e43234 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2f80953a600947b79f5e500d1cb36f09]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:24.475360 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6faeb5cba6044a46907cc82448e43234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.475787 17185 tablet_bootstrap.cc:654] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:24.477819 17187 tablet_bootstrap.cc:492] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:24.479542 17187 tablet_bootstrap.cc:654] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:24.481359 17187 tablet_bootstrap.cc:492] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:24.481492 17187 ts_tablet_manager.cc:1367] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:24.482198 17187 raft_consensus.cc:348] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:24.482414 17187 raft_consensus.cc:374] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:24.482579 17187 raft_consensus.cc:729] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.482573 12970 tablet_service.cc:1467] Processing CreateTablet for tablet f50084f1f7784cbfbcb5cb338dbd192e (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2f80953a600947b79f5e500d1cb36f09]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:24.482811 17187 consensus_queue.cc:260] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:24.482954 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50084f1f7784cbfbcb5cb338dbd192e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.483320 17187 ts_tablet_manager.cc:1398] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:24.487344 15468 tablet_service.cc:1467] Processing CreateTablet for tablet f50084f1f7784cbfbcb5cb338dbd192e (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=2f80953a600947b79f5e500d1cb36f09]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:24.487752 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50084f1f7784cbfbcb5cb338dbd192e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.494369 17191 tablet_bootstrap.cc:492] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:24.497711 17191 tablet_bootstrap.cc:654] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:24.498260 17192 tablet_bootstrap.cc:492] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:24.504825 17192 tablet_bootstrap.cc:654] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:24.505445 12579 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:24.505645 12579 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:24.506199 17186 tablet_bootstrap.cc:492] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:24.506419 17185 tablet_bootstrap.cc:492] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:24.506582 17185 ts_tablet_manager.cc:1367] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.012s
I20240502 08:18:24.507838 17186 tablet_bootstrap.cc:654] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:24.508006 17185 raft_consensus.cc:348] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:24.508242 17185 raft_consensus.cc:374] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:24.508375 17185 raft_consensus.cc:729] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.508628 17185 consensus_queue.cc:260] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:24.509238 17185 ts_tablet_manager.cc:1398] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:24.509479 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 9f1e3b48fdd14c6c859aa31ac990a17d
I20240502 08:18:24.509652 17194 tablet_bootstrap.cc:492] T 6faeb5cba6044a46907cc82448e43234 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:24.510933 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1e3b48fdd14c6c859aa31ac990a17d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:24 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:24.510933 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1e3b48fdd14c6c859aa31ac990a17d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:24 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:24.511096 17191 tablet_bootstrap.cc:492] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:24.511232 17191 ts_tablet_manager.cc:1367] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20240502 08:18:24.512347 17191 raft_consensus.cc:348] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:24.512615 17191 raft_consensus.cc:374] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:24.512756 17191 raft_consensus.cc:729] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.513056 17191 consensus_queue.cc:260] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:24.513947 17191 ts_tablet_manager.cc:1398] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:24.519235 17195 tablet_replica.cc:330] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:24.520435 17195 raft_consensus.cc:2229] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:24.520756 17195 raft_consensus.cc:2258] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:24.521548 17195 ts_tablet_manager.cc:1875] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:24.522399 17190 tablet_bootstrap.cc:492] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:24.510645 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1e3b48fdd14c6c859aa31ac990a17d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:24 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:24.523100 17198 tablet_replica.cc:330] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:24.523362 17198 raft_consensus.cc:2229] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:24.523684 17198 raft_consensus.cc:2258] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:24.523921 17190 tablet_bootstrap.cc:654] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:24.524580 17198 ts_tablet_manager.cc:1875] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:24.524834 17194 tablet_bootstrap.cc:654] T 6faeb5cba6044a46907cc82448e43234 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:24.526674 17196 tablet_replica.cc:330] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:24.529008 17196 raft_consensus.cc:2229] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:24.530121 17196 raft_consensus.cc:2258] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:24.531046 17196 ts_tablet_manager.cc:1875] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:24.531057 17198 ts_tablet_manager.cc:1888] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:24.531158 17198 log.cc:1196] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/9f1e3b48fdd14c6c859aa31ac990a17d
I20240502 08:18:24.531633 17198 ts_tablet_manager.cc:1909] T 9f1e3b48fdd14c6c859aa31ac990a17d P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:24.532796 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 9f1e3b48fdd14c6c859aa31ac990a17d (table rebalancer_dynamic_table_new_name [id=fbb936619eb54c00aebfd01210f7e41a]) successfully deleted
I20240502 08:18:24.533407 17195 ts_tablet_manager.cc:1888] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:24.533532 17195 log.cc:1196] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/9f1e3b48fdd14c6c859aa31ac990a17d
I20240502 08:18:24.533988 17195 ts_tablet_manager.cc:1909] T 9f1e3b48fdd14c6c859aa31ac990a17d P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:24.535125 17190 tablet_bootstrap.cc:492] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:24.535316 17190 ts_tablet_manager.cc:1367] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240502 08:18:24.535455 17194 tablet_bootstrap.cc:492] T 6faeb5cba6044a46907cc82448e43234 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:24.535663 17194 ts_tablet_manager.cc:1367] T 6faeb5cba6044a46907cc82448e43234 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240502 08:18:24.535993 17186 tablet_bootstrap.cc:492] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:24.536183 17186 ts_tablet_manager.cc:1367] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.003s
I20240502 08:18:24.536432 17190 raft_consensus.cc:348] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:24.536459 17194 raft_consensus.cc:348] T 6faeb5cba6044a46907cc82448e43234 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:24.536675 17190 raft_consensus.cc:374] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:24.536963 17190 raft_consensus.cc:729] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.537055 17194 raft_consensus.cc:374] T 6faeb5cba6044a46907cc82448e43234 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:24.537047 17186 raft_consensus.cc:348] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:24.537240 17186 raft_consensus.cc:374] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:24.537374 17186 raft_consensus.cc:729] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.537354 17190 consensus_queue.cc:260] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:24.537467 17194 raft_consensus.cc:729] T 6faeb5cba6044a46907cc82448e43234 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.537900 17190 ts_tablet_manager.cc:1398] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:24.538228 17194 consensus_queue.cc:260] T 6faeb5cba6044a46907cc82448e43234 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:24.538820 17186 consensus_queue.cc:260] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:24.539363 17192 tablet_bootstrap.cc:492] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:24.539594 17192 ts_tablet_manager.cc:1367] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20240502 08:18:24.539882 16964 raft_consensus.cc:482] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:24.540046 16964 raft_consensus.cc:504] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:24.540314 17186 ts_tablet_manager.cc:1398] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:18:24.540773 16964 leader_election.cc:288] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:24.541306 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0a5745bdba436e848ba33029ef01b4" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:24.543821 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 9f1e3b48fdd14c6c859aa31ac990a17d (table rebalancer_dynamic_table_new_name [id=fbb936619eb54c00aebfd01210f7e41a]) successfully deleted
W20240502 08:18:24.545205 13055 leader_election.cc:341] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:24.547969 17192 raft_consensus.cc:348] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:24.548206 17192 raft_consensus.cc:374] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:24.548312 17192 raft_consensus.cc:729] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.548537 17192 consensus_queue.cc:260] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:24.549165 17192 ts_tablet_manager.cc:1398] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 08:18:24.549489 17192 tablet_bootstrap.cc:492] T 6faeb5cba6044a46907cc82448e43234 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:24.551254 17192 tablet_bootstrap.cc:654] T 6faeb5cba6044a46907cc82448e43234 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:24.551270 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0a5745bdba436e848ba33029ef01b4" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:24.551507 15621 raft_consensus.cc:2454] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:24.552028 13058 leader_election.cc:302] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:24.552484 16964 raft_consensus.cc:2781] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:24.552621 16964 raft_consensus.cc:482] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:24.552716 16964 raft_consensus.cc:3037] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.553580 17194 ts_tablet_manager.cc:1398] T 6faeb5cba6044a46907cc82448e43234 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 08:18:24.555477 16964 raft_consensus.cc:504] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:24.556149 16964 leader_election.cc:288] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:24.556576 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0a5745bdba436e848ba33029ef01b4" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:24.556672 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0a5745bdba436e848ba33029ef01b4" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:24.556772 15754 raft_consensus.cc:3037] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.556869 15620 raft_consensus.cc:3037] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.559274 15754 raft_consensus.cc:2454] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:24.559305 15620 raft_consensus.cc:2454] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:24.559953 17196 ts_tablet_manager.cc:1888] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:24.560091 17196 log.cc:1196] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/9f1e3b48fdd14c6c859aa31ac990a17d
I20240502 08:18:24.559973 13055 leader_election.cc:302] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:24.560550 17196 ts_tablet_manager.cc:1909] T 9f1e3b48fdd14c6c859aa31ac990a17d P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:24.560930 16964 raft_consensus.cc:2781] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:24.561056 16964 raft_consensus.cc:686] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:18:24.561352 16964 consensus_queue.cc:237] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:24.561846 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 9f1e3b48fdd14c6c859aa31ac990a17d (table rebalancer_dynamic_table_new_name [id=fbb936619eb54c00aebfd01210f7e41a]) successfully deleted
I20240502 08:18:24.562969 17192 tablet_bootstrap.cc:492] T 6faeb5cba6044a46907cc82448e43234 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:24.563305 17192 ts_tablet_manager.cc:1367] T 6faeb5cba6044a46907cc82448e43234 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 08:18:24.564203 17192 raft_consensus.cc:348] T 6faeb5cba6044a46907cc82448e43234 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:24.564517 17192 raft_consensus.cc:374] T 6faeb5cba6044a46907cc82448e43234 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:24.564714 17192 raft_consensus.cc:729] T 6faeb5cba6044a46907cc82448e43234 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.565043 17192 consensus_queue.cc:260] T 6faeb5cba6044a46907cc82448e43234 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:24.569382 17192 ts_tablet_manager.cc:1398] T 6faeb5cba6044a46907cc82448e43234 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:24.569751 17192 tablet_bootstrap.cc:492] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:24.571867 17192 tablet_bootstrap.cc:654] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:24.571892 17064 raft_consensus.cc:482] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:24.572055 17064 raft_consensus.cc:504] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:24.572715 17064 leader_election.cc:288] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:24.573441 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50084f1f7784cbfbcb5cb338dbd192e" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:24.573678 15487 raft_consensus.cc:2454] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:24.574254 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50084f1f7784cbfbcb5cb338dbd192e" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:24.574400 12926 leader_election.cc:302] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:24.574643 17064 raft_consensus.cc:2781] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:24.574754 17064 raft_consensus.cc:482] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:24.574846 17064 raft_consensus.cc:3037] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
W20240502 08:18:24.575428 12925 leader_election.cc:341] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:24.577358 17064 raft_consensus.cc:504] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:24.578405 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50084f1f7784cbfbcb5cb338dbd192e" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
W20240502 08:18:24.578895 12925 leader_election.cc:341] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:24.578982 17064 leader_election.cc:288] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:24.579438 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50084f1f7784cbfbcb5cb338dbd192e" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:24.579648 15486 raft_consensus.cc:3037] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.581722 15486 raft_consensus.cc:2454] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:24.582064 12926 leader_election.cc:302] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters: 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:24.582679 17064 raft_consensus.cc:2781] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:24.582827 17064 raft_consensus.cc:686] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:24.583127 17064 consensus_queue.cc:237] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:24.587227 17192 tablet_bootstrap.cc:492] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:24.587390 17192 ts_tablet_manager.cc:1367] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20240502 08:18:24.587471 12584 catalog_manager.cc:5484] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:24.588086 17192 raft_consensus.cc:348] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:24.588269 17192 raft_consensus.cc:374] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:24.588366 17192 raft_consensus.cc:729] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.588569 17192 consensus_queue.cc:260] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:24.589018 17192 ts_tablet_manager.cc:1398] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:24.591176 12579 catalog_manager.cc:5484] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:24.602358 16854 raft_consensus.cc:482] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:24.602519 16854 raft_consensus.cc:504] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:24.603149 16854 leader_election.cc:288] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:24.603569 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50084f1f7784cbfbcb5cb338dbd192e" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:24.603924 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50084f1f7784cbfbcb5cb338dbd192e" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:24.604146 15488 raft_consensus.cc:2379] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9333cb10227b4801b4fa7c24e22e131b in current term 1: Already voted for candidate 60b6e265e7754b7187307d05ef405dd4 in this term.
I20240502 08:18:24.604516 15688 leader_election.cc:302] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b; no voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:24.604796 16854 raft_consensus.cc:3037] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.606573 16854 raft_consensus.cc:2726] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:24.611704 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:24.619066 15866 tablet_service.cc:1467] Processing CreateTablet for tablet cbcef97227a245eba142ef728fed2cc8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=38e9636e2f824edf97af0d2d6e188b78]), partition=
I20240502 08:18:24.619071 13101 tablet_service.cc:1467] Processing CreateTablet for tablet cbcef97227a245eba142ef728fed2cc8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=38e9636e2f824edf97af0d2d6e188b78]), partition=
I20240502 08:18:24.619385 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbcef97227a245eba142ef728fed2cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.619398 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbcef97227a245eba142ef728fed2cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.623428 17187 tablet_bootstrap.cc:492] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:24.624909 17187 tablet_bootstrap.cc:654] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:24.625692 15733 tablet_service.cc:1467] Processing CreateTablet for tablet cbcef97227a245eba142ef728fed2cc8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=38e9636e2f824edf97af0d2d6e188b78]), partition=
I20240502 08:18:24.626102 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbcef97227a245eba142ef728fed2cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:24.629032 17186 tablet_bootstrap.cc:492] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:24.630296 17186 tablet_bootstrap.cc:654] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:24.640595 17192 tablet_bootstrap.cc:492] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:24.642168 17192 tablet_bootstrap.cc:654] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:24.643555 17186 tablet_bootstrap.cc:492] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:24.643718 17186 ts_tablet_manager.cc:1367] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20240502 08:18:24.644289 17187 tablet_bootstrap.cc:492] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:24.644444 17187 ts_tablet_manager.cc:1367] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.008s
I20240502 08:18:24.644475 17186 raft_consensus.cc:348] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:24.644694 17186 raft_consensus.cc:374] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:24.644764 17186 raft_consensus.cc:729] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.644975 17186 consensus_queue.cc:260] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:24.645493 17186 ts_tablet_manager.cc:1398] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:24.647763 17187 raft_consensus.cc:348] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:24.648002 17187 raft_consensus.cc:374] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:24.648079 17187 raft_consensus.cc:729] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.648306 17187 consensus_queue.cc:260] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:24.648878 17187 ts_tablet_manager.cc:1398] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:18:24.662585 17192 tablet_bootstrap.cc:492] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:24.663467 17192 ts_tablet_manager.cc:1367] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.009s
I20240502 08:18:24.664414 17192 raft_consensus.cc:348] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:24.664610 17192 raft_consensus.cc:374] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:24.664685 17192 raft_consensus.cc:729] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:24.664988 17192 consensus_queue.cc:260] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:24.666090 17192 ts_tablet_manager.cc:1398] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:24.759689 16964 raft_consensus.cc:482] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:24.759883 16964 raft_consensus.cc:504] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:24.760673 16964 leader_election.cc:288] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:24.761178 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcef97227a245eba142ef728fed2cc8" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:24.761410 15754 raft_consensus.cc:2454] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:24.761842 13055 leader_election.cc:302] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:24.762113 16964 raft_consensus.cc:2781] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:24.762210 16964 raft_consensus.cc:482] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:24.762342 16964 raft_consensus.cc:3037] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.764580 16964 raft_consensus.cc:504] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:24.765342 16964 leader_election.cc:288] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:24.765597 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcef97227a245eba142ef728fed2cc8" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:24.765802 15754 raft_consensus.cc:3037] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.767598 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcef97227a245eba142ef728fed2cc8" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:24.767676 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcef97227a245eba142ef728fed2cc8" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:24.767861 15754 raft_consensus.cc:2454] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:24.767867 15887 raft_consensus.cc:2454] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:24.768216 13055 leader_election.cc:302] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 9333cb10227b4801b4fa7c24e22e131b; no voters: 10f0e36203914b85b77e78609ffced98
I20240502 08:18:24.768455 16964 raft_consensus.cc:2781] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:24.768656 16964 raft_consensus.cc:686] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:18:24.768942 16964 consensus_queue.cc:237] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:24.772505 12584 catalog_manager.cc:5484] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:24.815804 16865 raft_consensus.cc:482] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:24.816005 16865 raft_consensus.cc:504] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:24.816644 16865 leader_election.cc:288] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:24.816991 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6faeb5cba6044a46907cc82448e43234" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:24.817201 15754 raft_consensus.cc:2454] T 6faeb5cba6044a46907cc82448e43234 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:24.817538 15818 leader_election.cc:302] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:24.817579 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6faeb5cba6044a46907cc82448e43234" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:24.817783 16865 raft_consensus.cc:2781] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:24.817786 12859 raft_consensus.cc:2454] T 6faeb5cba6044a46907cc82448e43234 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:24.817869 16865 raft_consensus.cc:482] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:24.817970 16865 raft_consensus.cc:3037] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.820061 16865 raft_consensus.cc:504] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:24.820649 16865 leader_election.cc:288] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:24.821043 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6faeb5cba6044a46907cc82448e43234" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:24.821215 15754 raft_consensus.cc:3037] T 6faeb5cba6044a46907cc82448e43234 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.821156 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6faeb5cba6044a46907cc82448e43234" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:24.821353 12859 raft_consensus.cc:3037] T 6faeb5cba6044a46907cc82448e43234 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.824009 12859 raft_consensus.cc:2454] T 6faeb5cba6044a46907cc82448e43234 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:24.824153 15754 raft_consensus.cc:2454] T 6faeb5cba6044a46907cc82448e43234 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:24.824450 15819 leader_election.cc:302] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:24.824833 16865 raft_consensus.cc:2781] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:24.825059 16865 raft_consensus.cc:686] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:24.825304 16865 consensus_queue.cc:237] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:24.828837 12584 catalog_manager.cc:5484] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:24.891222 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:24.896955 15887 raft_consensus.cc:3037] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:24.896998 15754 raft_consensus.cc:1261] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:24.897756 16964 consensus_queue.cc:1035] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:24.899953 15887 raft_consensus.cc:1261] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:24.900419 16964 consensus_queue.cc:1035] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:24.901741 17220 tablet.cc:1722] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:24.902069 12584 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet cbcef97227a245eba142ef728fed2cc8 (table rebalancer_dynamic_table [id=38e9636e2f824edf97af0d2d6e188b78]). Expected version 1 got 0
I20240502 08:18:24.902791 17220 tablet.cc:1620] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:24.909162 17221 tablet.cc:1722] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:24.912245 17221 tablet.cc:1620] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:24.912395 12579 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet cbcef97227a245eba142ef728fed2cc8 (table rebalancer_dynamic_table [id=38e9636e2f824edf97af0d2d6e188b78]). Expected version 1 got 0
I20240502 08:18:24.912371 17223 tablet.cc:1722] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:24.913007 12583 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet cbcef97227a245eba142ef728fed2cc8 (table rebalancer_dynamic_table [id=38e9636e2f824edf97af0d2d6e188b78]). Expected version 1 got 0
I20240502 08:18:24.913643 17223 tablet.cc:1620] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:24.918865 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=38e9636e2f824edf97af0d2d6e188b78] alter complete (version 1)
I20240502 08:18:24.961789 17205 consensus_queue.cc:1035] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:24.966989 16964 consensus_queue.cc:1035] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:25.023612 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:25.031278 17220 tablet.cc:1722] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:25.031729 17220 tablet.cc:1620] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:25.031841 17221 tablet.cc:1722] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:25.032312 17221 tablet.cc:1620] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:25.034973 17223 tablet.cc:1722] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:25.035436 17223 tablet.cc:1620] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:25.042268 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=38e9636e2f824edf97af0d2d6e188b78] alter complete (version 2)
I20240502 08:18:25.091809 17105 consensus_queue.cc:1035] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:25.095697 17105 consensus_queue.cc:1035] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:25.144729 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:25.152253 17220 tablet.cc:1722] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:25.153355 17221 tablet.cc:1722] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:25.153967 17223 tablet.cc:1722] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:25.162825 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=38e9636e2f824edf97af0d2d6e188b78] alter complete (version 3)
I20240502 08:18:25.256214 16865 consensus_queue.cc:1035] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:25.260426 16632 consensus_queue.cc:1035] T 6faeb5cba6044a46907cc82448e43234 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:25.265733 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:25.265916 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:25.269016 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet cbcef97227a245eba142ef728fed2cc8
I20240502 08:18:25.269779 15866 tablet_service.cc:1514] Processing DeleteTablet for tablet cbcef97227a245eba142ef728fed2cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:25 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:25.269783 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet cbcef97227a245eba142ef728fed2cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:25 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:25.270579 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet cbcef97227a245eba142ef728fed2cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:25 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:25.272085 17238 tablet_replica.cc:330] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:25.272279 17238 raft_consensus.cc:2229] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:25.272574 17238 raft_consensus.cc:2258] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:25.273247 17238 ts_tablet_manager.cc:1875] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:25.281121 17238 ts_tablet_manager.cc:1888] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:25.281416 17238 log.cc:1196] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/cbcef97227a245eba142ef728fed2cc8
I20240502 08:18:25.281863 17238 ts_tablet_manager.cc:1909] T cbcef97227a245eba142ef728fed2cc8 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:25.283514 17236 tablet_replica.cc:330] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:25.283715 17236 raft_consensus.cc:2229] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:25.283936 17236 raft_consensus.cc:2258] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:25.283942 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet cbcef97227a245eba142ef728fed2cc8 (table rebalancer_dynamic_table_new_name [id=38e9636e2f824edf97af0d2d6e188b78]) successfully deleted
I20240502 08:18:25.284483 17237 tablet_replica.cc:330] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:25.284691 17237 raft_consensus.cc:2229] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:25.284875 17236 ts_tablet_manager.cc:1875] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:25.284914 17237 raft_consensus.cc:2258] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:25.285624 17237 ts_tablet_manager.cc:1875] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:25.291337 17236 ts_tablet_manager.cc:1888] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:25.291447 17236 log.cc:1196] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/cbcef97227a245eba142ef728fed2cc8
I20240502 08:18:25.291839 17236 ts_tablet_manager.cc:1909] T cbcef97227a245eba142ef728fed2cc8 P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:25.292416 17237 ts_tablet_manager.cc:1888] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:25.292536 17237 log.cc:1196] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/cbcef97227a245eba142ef728fed2cc8
I20240502 08:18:25.292866 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet cbcef97227a245eba142ef728fed2cc8 (table rebalancer_dynamic_table_new_name [id=38e9636e2f824edf97af0d2d6e188b78]) successfully deleted
I20240502 08:18:25.292923 17237 ts_tablet_manager.cc:1909] T cbcef97227a245eba142ef728fed2cc8 P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:25.294005 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet cbcef97227a245eba142ef728fed2cc8 (table rebalancer_dynamic_table_new_name [id=38e9636e2f824edf97af0d2d6e188b78]) successfully deleted
I20240502 08:18:25.367238 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:25.371528 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:25.376307 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 5fdc47843a2c42b2b1713b09acb629a7 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6eff13bc8a8247babea96b8f0537a4cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:25.376761 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdc47843a2c42b2b1713b09acb629a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.376909 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 5fdc47843a2c42b2b1713b09acb629a7 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6eff13bc8a8247babea96b8f0537a4cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:25.377296 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdc47843a2c42b2b1713b09acb629a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.377429 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 8de012363e4f4455acd96cb674ab0c4b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6eff13bc8a8247babea96b8f0537a4cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:25.377863 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de012363e4f4455acd96cb674ab0c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.379737 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 5fdc47843a2c42b2b1713b09acb629a7 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6eff13bc8a8247babea96b8f0537a4cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:25.379765 15866 tablet_service.cc:1467] Processing CreateTablet for tablet 8de012363e4f4455acd96cb674ab0c4b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6eff13bc8a8247babea96b8f0537a4cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:25.380160 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdc47843a2c42b2b1713b09acb629a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.380213 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de012363e4f4455acd96cb674ab0c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.382827 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 7dcaab7ded074ca38dda02c3bdf4c5e9 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6eff13bc8a8247babea96b8f0537a4cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:25.383208 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcaab7ded074ca38dda02c3bdf4c5e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.387012 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 7dcaab7ded074ca38dda02c3bdf4c5e9 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6eff13bc8a8247babea96b8f0537a4cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:25.387456 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcaab7ded074ca38dda02c3bdf4c5e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.398928 12710 tablet_service.cc:1467] Processing CreateTablet for tablet cae713542eac4d75996800c9c63387ae (DEFAULT_TABLE table=rebalancer_dynamic_table [id=657f2446d3ef41c4a2cb4aae286c7138]), partition=
I20240502 08:18:25.399286 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae713542eac4d75996800c9c63387ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.401175 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 7dcaab7ded074ca38dda02c3bdf4c5e9 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6eff13bc8a8247babea96b8f0537a4cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:25.401587 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 8de012363e4f4455acd96cb674ab0c4b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6eff13bc8a8247babea96b8f0537a4cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:25.401886 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcaab7ded074ca38dda02c3bdf4c5e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.402423 17244 tablet_bootstrap.cc:492] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:25.404639 17242 tablet_bootstrap.cc:492] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:25.405634 17244 tablet_bootstrap.cc:654] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:25.406244 17242 tablet_bootstrap.cc:654] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:25.406889 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de012363e4f4455acd96cb674ab0c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.406985 12841 tablet_service.cc:1467] Processing CreateTablet for tablet cae713542eac4d75996800c9c63387ae (DEFAULT_TABLE table=rebalancer_dynamic_table [id=657f2446d3ef41c4a2cb4aae286c7138]), partition=
I20240502 08:18:25.407348 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae713542eac4d75996800c9c63387ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.414577 15467 tablet_service.cc:1467] Processing CreateTablet for tablet cae713542eac4d75996800c9c63387ae (DEFAULT_TABLE table=rebalancer_dynamic_table [id=657f2446d3ef41c4a2cb4aae286c7138]), partition=
I20240502 08:18:25.414918 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae713542eac4d75996800c9c63387ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:25.420739 17247 tablet_bootstrap.cc:492] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:25.422480 17247 tablet_bootstrap.cc:654] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:25.424623 17244 tablet_bootstrap.cc:492] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:25.425343 17244 ts_tablet_manager.cc:1367] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.002s
I20240502 08:18:25.444358 17250 tablet_bootstrap.cc:492] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
W20240502 08:18:25.444710 12664 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 50799.2us
I20240502 08:18:25.446951 17244 raft_consensus.cc:348] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.447201 17244 raft_consensus.cc:374] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:25.447289 17244 raft_consensus.cc:729] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.451406 17251 tablet_bootstrap.cc:492] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:25.451802 17248 tablet_bootstrap.cc:492] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:25.452028 17249 tablet_bootstrap.cc:492] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:25.454811 17250 tablet_bootstrap.cc:654] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:25.454946 17244 consensus_queue.cc:260] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.458712 17247 tablet_bootstrap.cc:492] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:25.459635 17244 ts_tablet_manager.cc:1398] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20240502 08:18:25.460111 17247 ts_tablet_manager.cc:1367] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.002s
I20240502 08:18:25.464147 17247 raft_consensus.cc:348] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:25.467061 17247 raft_consensus.cc:374] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:25.467355 17247 raft_consensus.cc:729] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.467905 17247 consensus_queue.cc:260] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:25.469559 17247 ts_tablet_manager.cc:1398] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 08:18:25.470628 17247 tablet_bootstrap.cc:492] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:25.471774 17248 tablet_bootstrap.cc:654] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:25.476908 17247 tablet_bootstrap.cc:654] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:25.481096 17247 tablet_bootstrap.cc:492] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:25.481401 17247 ts_tablet_manager.cc:1367] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20240502 08:18:25.482270 17247 raft_consensus.cc:348] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:25.483321 17247 raft_consensus.cc:374] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:25.483594 17247 raft_consensus.cc:729] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.484040 17251 tablet_bootstrap.cc:654] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:25.484052 17247 consensus_queue.cc:260] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:25.484202 17249 tablet_bootstrap.cc:654] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:25.485924 17247 ts_tablet_manager.cc:1398] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:25.486617 17247 tablet_bootstrap.cc:492] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:25.488170 17241 tablet_bootstrap.cc:492] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:25.488454 17248 tablet_bootstrap.cc:492] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:25.488543 17251 tablet_bootstrap.cc:492] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:25.488749 17251 ts_tablet_manager.cc:1367] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20240502 08:18:25.489492 17242 tablet_bootstrap.cc:492] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:25.489509 17247 tablet_bootstrap.cc:654] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:25.489665 17242 ts_tablet_manager.cc:1367] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.085s user 0.023s sys 0.045s
I20240502 08:18:25.489650 17251 raft_consensus.cc:348] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.489996 17251 raft_consensus.cc:374] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:25.490142 17251 raft_consensus.cc:729] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.490551 17248 ts_tablet_manager.cc:1367] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240502 08:18:25.490420 17251 consensus_queue.cc:260] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.490595 17242 raft_consensus.cc:348] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:25.490809 17242 raft_consensus.cc:374] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:25.490906 17242 raft_consensus.cc:729] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.491245 17251 ts_tablet_manager.cc:1398] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:25.491245 17242 consensus_queue.cc:260] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:25.492034 17242 ts_tablet_manager.cc:1398] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 08:18:25.492348 17242 tablet_bootstrap.cc:492] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:25.492676 17250 tablet_bootstrap.cc:492] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:25.492898 17250 ts_tablet_manager.cc:1367] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.009s
I20240502 08:18:25.493741 17250 raft_consensus.cc:348] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.494041 17250 raft_consensus.cc:374] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:25.494175 17250 raft_consensus.cc:729] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.494518 17250 consensus_queue.cc:260] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.494763 17242 tablet_bootstrap.cc:654] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:25.496455 17250 ts_tablet_manager.cc:1398] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:25.497320 17249 tablet_bootstrap.cc:492] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:25.497473 17249 ts_tablet_manager.cc:1367] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20240502 08:18:25.498309 17249 raft_consensus.cc:348] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.498576 17249 raft_consensus.cc:374] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:25.498680 17249 raft_consensus.cc:729] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.498978 17249 consensus_queue.cc:260] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.499760 17249 ts_tablet_manager.cc:1398] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:25.500255 17249 tablet_bootstrap.cc:492] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:25.499948 17248 raft_consensus.cc:348] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:25.500643 17248 raft_consensus.cc:374] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:25.500752 17248 raft_consensus.cc:729] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.500967 17248 consensus_queue.cc:260] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:25.501569 17248 ts_tablet_manager.cc:1398] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 08:18:25.502144 17249 tablet_bootstrap.cc:654] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:25.507628 17241 tablet_bootstrap.cc:654] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:25.514178 17241 tablet_bootstrap.cc:492] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:25.514401 17241 ts_tablet_manager.cc:1367] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20240502 08:18:25.517441 17241 raft_consensus.cc:348] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:25.518636 17241 raft_consensus.cc:374] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:25.518936 17241 raft_consensus.cc:729] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.519493 17241 consensus_queue.cc:260] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:25.520684 17241 ts_tablet_manager.cc:1398] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 08:18:25.526082 17247 tablet_bootstrap.cc:492] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:25.526260 17247 ts_tablet_manager.cc:1367] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.040s user 0.019s sys 0.006s
I20240502 08:18:25.527040 17247 raft_consensus.cc:348] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.527240 17247 raft_consensus.cc:374] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:25.527344 17247 raft_consensus.cc:729] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.527781 17247 consensus_queue.cc:260] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.529350 17247 ts_tablet_manager.cc:1398] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:25.529552 17249 tablet_bootstrap.cc:492] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:25.529771 17249 ts_tablet_manager.cc:1367] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.014s
I20240502 08:18:25.530616 17249 raft_consensus.cc:348] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.530885 17249 raft_consensus.cc:374] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:25.531008 17249 raft_consensus.cc:729] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.531942 17242 tablet_bootstrap.cc:492] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:25.532117 17242 ts_tablet_manager.cc:1367] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.007s
I20240502 08:18:25.531297 17249 consensus_queue.cc:260] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.533231 17249 ts_tablet_manager.cc:1398] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:25.532832 17242 raft_consensus.cc:348] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:25.534158 17242 raft_consensus.cc:374] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:25.534236 17242 raft_consensus.cc:729] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:25.534477 17242 consensus_queue.cc:260] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:25.535079 17242 ts_tablet_manager.cc:1398] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:25.551414 17154 raft_consensus.cc:482] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:25.551541 17154 raft_consensus.cc:504] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.552160 17154 leader_election.cc:288] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:25.552582 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae713542eac4d75996800c9c63387ae" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:25.552661 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae713542eac4d75996800c9c63387ae" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:25.552827 12859 raft_consensus.cc:2454] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:25.552827 15488 raft_consensus.cc:2454] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:25.553292 12664 leader_election.cc:302] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:25.553547 17154 raft_consensus.cc:2781] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:25.553695 17154 raft_consensus.cc:482] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:25.553788 17154 raft_consensus.cc:3037] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:25.557395 17154 raft_consensus.cc:504] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.557879 17154 leader_election.cc:288] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:25.558257 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae713542eac4d75996800c9c63387ae" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:25.558320 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cae713542eac4d75996800c9c63387ae" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:25.558427 12859 raft_consensus.cc:3037] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:25.558559 15488 raft_consensus.cc:3037] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:25.560236 12859 raft_consensus.cc:2454] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:25.560331 15488 raft_consensus.cc:2454] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:25.560575 12664 leader_election.cc:302] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:25.560829 17154 raft_consensus.cc:2781] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:25.560989 17154 raft_consensus.cc:686] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:18:25.561254 17154 consensus_queue.cc:237] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.564875 12579 catalog_manager.cc:5484] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:25.624255 16964 raft_consensus.cc:482] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:25.624450 16964 raft_consensus.cc:504] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:25.625216 16964 leader_election.cc:288] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:25.625631 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcaab7ded074ca38dda02c3bdf4c5e9" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:25.625762 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcaab7ded074ca38dda02c3bdf4c5e9" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:25.625898 15887 raft_consensus.cc:2454] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:25.625962 15754 raft_consensus.cc:2454] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:25.626368 13057 leader_election.cc:302] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:25.626662 16964 raft_consensus.cc:2781] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:25.626757 16964 raft_consensus.cc:482] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:25.626870 16964 raft_consensus.cc:3037] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:25.629524 16964 raft_consensus.cc:504] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:25.630019 16964 leader_election.cc:288] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:25.630386 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcaab7ded074ca38dda02c3bdf4c5e9" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:25.630571 15887 raft_consensus.cc:3037] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:25.630610 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dcaab7ded074ca38dda02c3bdf4c5e9" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:25.630815 15754 raft_consensus.cc:3037] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:25.633265 15887 raft_consensus.cc:2454] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:25.633355 15754 raft_consensus.cc:2454] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:25.633673 13057 leader_election.cc:302] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:25.633940 16964 raft_consensus.cc:2781] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:25.634088 16964 raft_consensus.cc:686] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:18:25.634326 16964 consensus_queue.cc:237] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:25.637773 12579 catalog_manager.cc:5484] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:25.694306 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:25.700063 15486 raft_consensus.cc:1261] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:25.700085 12859 raft_consensus.cc:1261] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:25.700657 17153 consensus_queue.cc:1035] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:25.701263 17154 consensus_queue.cc:1035] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:25.705781 12584 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet cae713542eac4d75996800c9c63387ae (table rebalancer_dynamic_table [id=657f2446d3ef41c4a2cb4aae286c7138]). Expected version 1 got 0
I20240502 08:18:25.706534 12579 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet cae713542eac4d75996800c9c63387ae (table rebalancer_dynamic_table [id=657f2446d3ef41c4a2cb4aae286c7138]). Expected version 1 got 0
I20240502 08:18:25.709290 17269 tablet.cc:1722] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:25.710304 17271 tablet.cc:1722] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:25.711072 17271 tablet.cc:1620] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:25.712769 12579 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet cae713542eac4d75996800c9c63387ae (table rebalancer_dynamic_table [id=657f2446d3ef41c4a2cb4aae286c7138]). Expected version 1 got 0
I20240502 08:18:25.713465 17007 raft_consensus.cc:482] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:25.713615 17007 raft_consensus.cc:504] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:25.714234 17007 leader_election.cc:288] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:25.714581 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de012363e4f4455acd96cb674ab0c4b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:25.714809 15755 raft_consensus.cc:2454] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:25.714767 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de012363e4f4455acd96cb674ab0c4b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:25.714972 15887 raft_consensus.cc:2454] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:25.715198 15553 leader_election.cc:302] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:25.715617 17007 raft_consensus.cc:2781] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:25.715729 17007 raft_consensus.cc:482] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:25.715807 17007 raft_consensus.cc:3037] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:25.716255 17269 tablet.cc:1620] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:25.718767 17007 raft_consensus.cc:504] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:25.719700 17007 leader_election.cc:288] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:25.719784 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de012363e4f4455acd96cb674ab0c4b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:25.719990 15887 raft_consensus.cc:3037] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:25.720203 17270 tablet.cc:1722] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:25.720582 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de012363e4f4455acd96cb674ab0c4b" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:25.720770 15755 raft_consensus.cc:3037] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:25.720825 17270 tablet.cc:1620] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:25.722442 15887 raft_consensus.cc:2454] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:25.723500 15555 leader_election.cc:302] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:25.724083 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=657f2446d3ef41c4a2cb4aae286c7138] alter complete (version 1)
I20240502 08:18:25.724428 17007 raft_consensus.cc:2781] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:25.724648 17007 raft_consensus.cc:686] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:25.724951 17007 consensus_queue.cc:237] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:25.729044 15755 raft_consensus.cc:2454] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:25.732386 12579 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet cae713542eac4d75996800c9c63387ae (table rebalancer_dynamic_table [id=657f2446d3ef41c4a2cb4aae286c7138]). Expected version 1 got 0
I20240502 08:18:25.737023 12584 catalog_manager.cc:5484] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:25.740149 17105 raft_consensus.cc:482] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:25.740306 17105 raft_consensus.cc:504] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.740960 17105 leader_election.cc:288] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:25.741240 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdc47843a2c42b2b1713b09acb629a7" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:25.741473 15755 raft_consensus.cc:2454] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:25.742249 12925 leader_election.cc:302] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:25.742548 17105 raft_consensus.cc:2781] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:25.742656 17105 raft_consensus.cc:482] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:25.742758 17105 raft_consensus.cc:3037] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:25.742707 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdc47843a2c42b2b1713b09acb629a7" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:25.742883 15486 raft_consensus.cc:2454] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:25.744849 17105 raft_consensus.cc:504] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.745338 17105 leader_election.cc:288] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:25.745740 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdc47843a2c42b2b1713b09acb629a7" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:25.745781 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fdc47843a2c42b2b1713b09acb629a7" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:25.745949 15755 raft_consensus.cc:3037] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:25.746099 15486 raft_consensus.cc:3037] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:25.748028 15755 raft_consensus.cc:2454] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:25.748172 15486 raft_consensus.cc:2454] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:25.748337 12925 leader_election.cc:302] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:25.748612 17105 raft_consensus.cc:2781] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:25.748796 17105 raft_consensus.cc:686] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:25.749073 17105 consensus_queue.cc:237] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:25.752271 12584 catalog_manager.cc:5484] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:25.834179 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:25.841503 17269 tablet.cc:1722] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:25.841961 17269 tablet.cc:1620] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:25.842501 17271 tablet.cc:1722] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:25.842974 17271 tablet.cc:1620] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:25.845768 17270 tablet.cc:1722] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:25.846218 17270 tablet.cc:1620] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:25.851219 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=657f2446d3ef41c4a2cb4aae286c7138] alter complete (version 2)
I20240502 08:18:25.955650 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:25.962127 17269 tablet.cc:1722] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:25.962627 17270 tablet.cc:1722] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:25.966109 17271 tablet.cc:1722] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:25.972342 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=657f2446d3ef41c4a2cb4aae286c7138] alter complete (version 3)
I20240502 08:18:26.075228 12579 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:26.075472 12579 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:26.078408 12579 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet cae713542eac4d75996800c9c63387ae
I20240502 08:18:26.079202 12710 tablet_service.cc:1514] Processing DeleteTablet for tablet cae713542eac4d75996800c9c63387ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:26 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:26.079272 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet cae713542eac4d75996800c9c63387ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:26 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:26.079300 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet cae713542eac4d75996800c9c63387ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:26 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:26.079874 17276 tablet_replica.cc:330] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:26.080089 17277 tablet_replica.cc:330] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:26.080189 17276 raft_consensus.cc:2229] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:26.080350 17277 raft_consensus.cc:2229] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:26.080420 17278 tablet_replica.cc:330] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:26.080638 17277 raft_consensus.cc:2258] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:26.080673 17278 raft_consensus.cc:2229] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:26.080677 17276 raft_consensus.cc:2258] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:26.081140 17278 raft_consensus.cc:2258] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:26.081832 17277 ts_tablet_manager.cc:1875] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:26.081960 17276 ts_tablet_manager.cc:1875] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:26.083031 17278 ts_tablet_manager.cc:1875] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:26.088111 17276 ts_tablet_manager.cc:1888] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:26.088233 17276 log.cc:1196] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/cae713542eac4d75996800c9c63387ae
I20240502 08:18:26.088641 17276 ts_tablet_manager.cc:1909] T cae713542eac4d75996800c9c63387ae P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:26.088857 17278 ts_tablet_manager.cc:1888] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:26.088973 17278 log.cc:1196] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/cae713542eac4d75996800c9c63387ae
I20240502 08:18:26.089380 17278 ts_tablet_manager.cc:1909] T cae713542eac4d75996800c9c63387ae P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:26.089619 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet cae713542eac4d75996800c9c63387ae (table rebalancer_dynamic_table_new_name [id=657f2446d3ef41c4a2cb4aae286c7138]) successfully deleted
I20240502 08:18:26.090350 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet cae713542eac4d75996800c9c63387ae (table rebalancer_dynamic_table_new_name [id=657f2446d3ef41c4a2cb4aae286c7138]) successfully deleted
I20240502 08:18:26.093745 17277 ts_tablet_manager.cc:1888] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:26.093854 17277 log.cc:1196] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/cae713542eac4d75996800c9c63387ae
I20240502 08:18:26.094197 17277 ts_tablet_manager.cc:1909] T cae713542eac4d75996800c9c63387ae P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:26.095021 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet cae713542eac4d75996800c9c63387ae (table rebalancer_dynamic_table_new_name [id=657f2446d3ef41c4a2cb4aae286c7138]) successfully deleted
I20240502 08:18:26.156948 17007 consensus_queue.cc:1035] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:26.162221 17263 consensus_queue.cc:1035] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:26.163276 17205 consensus_queue.cc:1035] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:26.166292 17105 consensus_queue.cc:1035] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:26.176496 17105 consensus_queue.cc:1035] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:26.180827 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:26.193390 15599 tablet_service.cc:1467] Processing CreateTablet for tablet a0f8eb0abb574a3aa82848c1d60c77ab (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1bb3163584b0496499f130307583a3da]), partition=
I20240502 08:18:26.193792 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f8eb0abb574a3aa82848c1d60c77ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.193946 15733 tablet_service.cc:1467] Processing CreateTablet for tablet a0f8eb0abb574a3aa82848c1d60c77ab (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1bb3163584b0496499f130307583a3da]), partition=
I20240502 08:18:26.194284 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f8eb0abb574a3aa82848c1d60c77ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.199450 13101 tablet_service.cc:1467] Processing CreateTablet for tablet a0f8eb0abb574a3aa82848c1d60c77ab (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1bb3163584b0496499f130307583a3da]), partition=
I20240502 08:18:26.199790 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f8eb0abb574a3aa82848c1d60c77ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.211959 17290 tablet_bootstrap.cc:492] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:26.214157 17290 tablet_bootstrap.cc:654] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:26.215468 17291 tablet_bootstrap.cc:492] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:26.217181 17291 tablet_bootstrap.cc:654] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:26.223315 17292 tablet_bootstrap.cc:492] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:26.231788 17292 tablet_bootstrap.cc:654] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:26.235859 17290 tablet_bootstrap.cc:492] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:26.235994 17290 ts_tablet_manager.cc:1367] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.015s
I20240502 08:18:26.237484 17291 tablet_bootstrap.cc:492] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:26.237461 17290 raft_consensus.cc:348] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:26.237676 17290 raft_consensus.cc:374] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:26.237774 17290 raft_consensus.cc:729] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:26.237943 17291 ts_tablet_manager.cc:1367] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240502 08:18:26.238008 16964 consensus_queue.cc:1035] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:26.238870 17290 consensus_queue.cc:260] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:26.239429 17291 raft_consensus.cc:348] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:26.239660 17290 ts_tablet_manager.cc:1398] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:26.239720 17291 raft_consensus.cc:374] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:26.239899 17291 raft_consensus.cc:729] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:26.240149 17291 consensus_queue.cc:260] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:26.243609 17291 ts_tablet_manager.cc:1398] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:18:26.249739 17292 tablet_bootstrap.cc:492] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:26.249879 17292 ts_tablet_manager.cc:1367] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20240502 08:18:26.250886 17292 raft_consensus.cc:348] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:26.252242 17292 raft_consensus.cc:374] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:26.252496 17292 raft_consensus.cc:729] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:26.252893 17292 consensus_queue.cc:260] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:26.253686 17292 ts_tablet_manager.cc:1398] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:26.314684 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:26.325323 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 65fa5d6450d9471395bda1af8c8c7973 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7b4c833a5fb54b3e967ac5b77b0558d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:26.325851 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fa5d6450d9471395bda1af8c8c7973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.326454 12841 tablet_service.cc:1467] Processing CreateTablet for tablet c631a6dbc0d44f76abf1cf5a14ea4a6d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7b4c833a5fb54b3e967ac5b77b0558d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:26.326871 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c631a6dbc0d44f76abf1cf5a14ea4a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.330755 15866 tablet_service.cc:1467] Processing CreateTablet for tablet c9ad7bf820c84aae9a8875fb92f8e5d5 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7b4c833a5fb54b3e967ac5b77b0558d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:26.331048 17274 raft_consensus.cc:482] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:26.331148 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ad7bf820c84aae9a8875fb92f8e5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.331076 12710 tablet_service.cc:1467] Processing CreateTablet for tablet c631a6dbc0d44f76abf1cf5a14ea4a6d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7b4c833a5fb54b3e967ac5b77b0558d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:26.331197 17274 raft_consensus.cc:504] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:26.331480 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c631a6dbc0d44f76abf1cf5a14ea4a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.331898 17274 leader_election.cc:288] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:26.336182 12709 tablet_service.cc:1467] Processing CreateTablet for tablet c9ad7bf820c84aae9a8875fb92f8e5d5 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7b4c833a5fb54b3e967ac5b77b0558d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:26.337127 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ad7bf820c84aae9a8875fb92f8e5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.337435 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 65fa5d6450d9471395bda1af8c8c7973 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7b4c833a5fb54b3e967ac5b77b0558d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:26.337795 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fa5d6450d9471395bda1af8c8c7973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.339380 15467 tablet_service.cc:1467] Processing CreateTablet for tablet c9ad7bf820c84aae9a8875fb92f8e5d5 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7b4c833a5fb54b3e967ac5b77b0558d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:26.339792 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ad7bf820c84aae9a8875fb92f8e5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.340914 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f8eb0abb574a3aa82848c1d60c77ab" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:26.341125 13119 raft_consensus.cc:2454] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:26.341074 13101 tablet_service.cc:1467] Processing CreateTablet for tablet c631a6dbc0d44f76abf1cf5a14ea4a6d (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7b4c833a5fb54b3e967ac5b77b0558d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:26.341497 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c631a6dbc0d44f76abf1cf5a14ea4a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.341555 15687 leader_election.cc:302] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:26.341812 17274 raft_consensus.cc:2781] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:26.341899 17274 raft_consensus.cc:482] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:26.341972 17274 raft_consensus.cc:3037] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:26.344615 17274 raft_consensus.cc:504] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:26.345476 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f8eb0abb574a3aa82848c1d60c77ab" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:26.345729 15620 raft_consensus.cc:2454] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:26.345659 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 65fa5d6450d9471395bda1af8c8c7973 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=7b4c833a5fb54b3e967ac5b77b0558d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:26.345757 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f8eb0abb574a3aa82848c1d60c77ab" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:26.345943 15621 raft_consensus.cc:3037] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:26.346073 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fa5d6450d9471395bda1af8c8c7973. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:26.346441 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0f8eb0abb574a3aa82848c1d60c77ab" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:26.346622 13119 raft_consensus.cc:3037] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:26.348714 15621 raft_consensus.cc:2454] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:26.349362 15690 leader_election.cc:302] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:26.349551 17274 leader_election.cc:288] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:26.349668 17275 raft_consensus.cc:2781] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:26.349680 13119 raft_consensus.cc:2454] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:26.349797 17275 raft_consensus.cc:686] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:26.350081 17275 consensus_queue.cc:237] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:26.366852 12584 catalog_manager.cc:5484] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:26.370602 17292 tablet_bootstrap.cc:492] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:26.372272 17292 tablet_bootstrap.cc:654] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:26.376443 17299 tablet_bootstrap.cc:492] T 65fa5d6450d9471395bda1af8c8c7973 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:26.376693 17300 tablet_bootstrap.cc:492] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:26.377045 17302 tablet_bootstrap.cc:492] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:26.378609 17302 tablet_bootstrap.cc:654] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:26.378686 17300 tablet_bootstrap.cc:654] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:26.378919 17297 tablet_bootstrap.cc:492] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:26.380375 17299 tablet_bootstrap.cc:654] T 65fa5d6450d9471395bda1af8c8c7973 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:26.380658 17297 tablet_bootstrap.cc:654] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:26.380983 17301 tablet_bootstrap.cc:492] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:26.385136 17301 tablet_bootstrap.cc:654] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:26.392832 17292 tablet_bootstrap.cc:492] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:26.394860 17292 ts_tablet_manager.cc:1367] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.005s
I20240502 08:18:26.395643 17292 raft_consensus.cc:348] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.395849 17292 raft_consensus.cc:374] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:26.395960 17292 raft_consensus.cc:729] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:26.396225 17292 consensus_queue.cc:260] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.399672 17292 ts_tablet_manager.cc:1398] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:18:26.404058 17301 tablet_bootstrap.cc:492] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:26.404258 17301 ts_tablet_manager.cc:1367] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20240502 08:18:26.404445 17302 tablet_bootstrap.cc:492] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:26.404656 17302 ts_tablet_manager.cc:1367] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.004s
I20240502 08:18:26.404771 17299 tablet_bootstrap.cc:492] T 65fa5d6450d9471395bda1af8c8c7973 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:26.404909 17299 ts_tablet_manager.cc:1367] T 65fa5d6450d9471395bda1af8c8c7973 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.004s
I20240502 08:18:26.405007 17301 raft_consensus.cc:348] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.405262 17301 raft_consensus.cc:374] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:26.405367 17297 tablet_bootstrap.cc:492] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:26.405393 17301 raft_consensus.cc:729] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:26.405539 17297 ts_tablet_manager.cc:1367] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.014s
I20240502 08:18:26.405777 17302 raft_consensus.cc:348] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:26.406059 17302 raft_consensus.cc:374] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:26.406397 17300 tablet_bootstrap.cc:492] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:26.406410 17302 raft_consensus.cc:729] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:26.406611 17300 ts_tablet_manager.cc:1367] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.019s
I20240502 08:18:26.406792 17302 consensus_queue.cc:260] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:26.407255 17301 consensus_queue.cc:260] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.407502 17300 raft_consensus.cc:348] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.407895 17300 raft_consensus.cc:374] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:26.407985 17300 raft_consensus.cc:729] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:26.408221 17300 consensus_queue.cc:260] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.408643 17297 raft_consensus.cc:348] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.409008 17297 raft_consensus.cc:374] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:26.409144 17297 raft_consensus.cc:729] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:26.409252 17299 raft_consensus.cc:348] T 65fa5d6450d9471395bda1af8c8c7973 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:26.409495 17299 raft_consensus.cc:374] T 65fa5d6450d9471395bda1af8c8c7973 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:26.409617 17299 raft_consensus.cc:729] T 65fa5d6450d9471395bda1af8c8c7973 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:26.409418 17297 consensus_queue.cc:260] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.410681 17301 ts_tablet_manager.cc:1398] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:26.411350 17299 consensus_queue.cc:260] T 65fa5d6450d9471395bda1af8c8c7973 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:26.411952 17300 ts_tablet_manager.cc:1398] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:26.412079 17297 ts_tablet_manager.cc:1398] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 08:18:26.412338 17297 tablet_bootstrap.cc:492] T 65fa5d6450d9471395bda1af8c8c7973 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:26.412382 17302 ts_tablet_manager.cc:1398] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 08:18:26.415422 17301 tablet_bootstrap.cc:492] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:26.416961 17302 tablet_bootstrap.cc:492] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:26.417611 17301 tablet_bootstrap.cc:654] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:26.418503 17302 tablet_bootstrap.cc:654] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:26.422488 17299 ts_tablet_manager.cc:1398] T 65fa5d6450d9471395bda1af8c8c7973 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240502 08:18:26.422606 17297 tablet_bootstrap.cc:654] T 65fa5d6450d9471395bda1af8c8c7973 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:26.426479 17302 tablet_bootstrap.cc:492] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:26.426702 17302 ts_tablet_manager.cc:1367] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240502 08:18:26.427057 17301 tablet_bootstrap.cc:492] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:26.427501 17302 raft_consensus.cc:348] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.427685 17302 raft_consensus.cc:374] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:26.427764 17302 raft_consensus.cc:729] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:26.427973 17302 consensus_queue.cc:260] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.428408 17302 ts_tablet_manager.cc:1398] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:26.427269 17301 ts_tablet_manager.cc:1367] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240502 08:18:26.434479 17301 raft_consensus.cc:348] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.434751 17301 raft_consensus.cc:374] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:26.434859 17301 raft_consensus.cc:729] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:26.435110 17301 consensus_queue.cc:260] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.435674 17301 ts_tablet_manager.cc:1398] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:26.443650 17297 tablet_bootstrap.cc:492] T 65fa5d6450d9471395bda1af8c8c7973 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:26.443809 17297 ts_tablet_manager.cc:1367] T 65fa5d6450d9471395bda1af8c8c7973 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.008s
I20240502 08:18:26.444561 17297 raft_consensus.cc:348] T 65fa5d6450d9471395bda1af8c8c7973 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:26.444797 17297 raft_consensus.cc:374] T 65fa5d6450d9471395bda1af8c8c7973 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:26.444869 17297 raft_consensus.cc:729] T 65fa5d6450d9471395bda1af8c8c7973 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:26.445143 17297 consensus_queue.cc:260] T 65fa5d6450d9471395bda1af8c8c7973 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:26.446847 17297 ts_tablet_manager.cc:1398] T 65fa5d6450d9471395bda1af8c8c7973 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:18:26.497318 17025 raft_consensus.cc:482] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:26.497467 17025 raft_consensus.cc:504] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.498025 17025 leader_election.cc:288] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:26.498365 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c631a6dbc0d44f76abf1cf5a14ea4a6d" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:26.498594 13119 raft_consensus.cc:2454] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:26.498612 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c631a6dbc0d44f76abf1cf5a14ea4a6d" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:26.498845 12727 raft_consensus.cc:2454] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:26.498982 12795 leader_election.cc:302] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:26.499398 17025 raft_consensus.cc:2781] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:26.499519 17025 raft_consensus.cc:482] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:26.499636 17025 raft_consensus.cc:3037] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:26.501761 17025 raft_consensus.cc:504] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.502367 17025 leader_election.cc:288] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:26.502928 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c631a6dbc0d44f76abf1cf5a14ea4a6d" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:26.503110 13119 raft_consensus.cc:3037] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:26.503113 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c631a6dbc0d44f76abf1cf5a14ea4a6d" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:26.503314 12727 raft_consensus.cc:3037] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:26.505252 13119 raft_consensus.cc:2454] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:26.505409 12727 raft_consensus.cc:2454] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:26.505678 12795 leader_election.cc:302] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:26.506218 17025 raft_consensus.cc:2781] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:26.506606 17025 raft_consensus.cc:686] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:18:26.507057 17025 consensus_queue.cc:237] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.510622 12584 catalog_manager.cc:5484] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:26.509213 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:26.512007 17081 raft_consensus.cc:482] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:26.512172 17081 raft_consensus.cc:504] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.512885 17081 leader_election.cc:288] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:26.513792 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ad7bf820c84aae9a8875fb92f8e5d5" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:26.514014 15886 raft_consensus.cc:2454] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:26.513976 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ad7bf820c84aae9a8875fb92f8e5d5" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:26.514372 15420 leader_election.cc:302] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:26.514305 12727 raft_consensus.cc:2454] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:26.514664 17081 raft_consensus.cc:2781] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:26.514782 17081 raft_consensus.cc:482] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:26.514878 17081 raft_consensus.cc:3037] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:26.517293 17081 raft_consensus.cc:504] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.517937 17081 leader_election.cc:288] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:26.518222 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ad7bf820c84aae9a8875fb92f8e5d5" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:26.518443 15886 raft_consensus.cc:3037] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:26.518874 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ad7bf820c84aae9a8875fb92f8e5d5" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:26.519109 12727 raft_consensus.cc:3037] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:26.519289 15621 raft_consensus.cc:1261] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:26.519289 13121 raft_consensus.cc:1261] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:26.520025 17274 consensus_queue.cc:1035] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:26.520725 17275 consensus_queue.cc:1035] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:26.521037 15886 raft_consensus.cc:2454] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:26.521764 15420 leader_election.cc:302] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:26.522039 12727 raft_consensus.cc:2454] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:26.522823 17227 raft_consensus.cc:2781] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:26.522994 17227 raft_consensus.cc:686] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:26.523289 17227 consensus_queue.cc:237] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:26.529209 12583 catalog_manager.cc:5484] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:26.529918 12584 catalog_manager.cc:5576] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979) does not have the latest schema for tablet a0f8eb0abb574a3aa82848c1d60c77ab (table rebalancer_dynamic_table [id=1bb3163584b0496499f130307583a3da]). Expected version 1 got 0
I20240502 08:18:26.532373 17318 tablet.cc:1722] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:26.533139 17318 tablet.cc:1620] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:26.540395 17319 tablet.cc:1722] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:26.543354 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=1bb3163584b0496499f130307583a3da] alter complete (version 1)
I20240502 08:18:26.544000 12583 catalog_manager.cc:5576] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983) does not have the latest schema for tablet a0f8eb0abb574a3aa82848c1d60c77ab (table rebalancer_dynamic_table [id=1bb3163584b0496499f130307583a3da]). Expected version 1 got 0
I20240502 08:18:26.543996 17319 tablet.cc:1620] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:26.545617 17320 tablet.cc:1722] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:26.546563 17320 tablet.cc:1620] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:26.548483 12584 catalog_manager.cc:5576] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983) does not have the latest schema for tablet a0f8eb0abb574a3aa82848c1d60c77ab (table rebalancer_dynamic_table [id=1bb3163584b0496499f130307583a3da]). Expected version 1 got 0
I20240502 08:18:26.550379 12584 catalog_manager.cc:5576] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979) does not have the latest schema for tablet a0f8eb0abb574a3aa82848c1d60c77ab (table rebalancer_dynamic_table [id=1bb3163584b0496499f130307583a3da]). Expected version 1 got 0
I20240502 08:18:26.634636 16632 raft_consensus.cc:482] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:26.634811 16632 raft_consensus.cc:504] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:26.635438 16632 leader_election.cc:288] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:26.635794 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa5d6450d9471395bda1af8c8c7973" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:26.635883 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa5d6450d9471395bda1af8c8c7973" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:26.635995 12992 raft_consensus.cc:2454] T 65fa5d6450d9471395bda1af8c8c7973 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:26.636065 12859 raft_consensus.cc:2454] T 65fa5d6450d9471395bda1af8c8c7973 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:26.636461 15819 leader_election.cc:302] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:26.636839 16632 raft_consensus.cc:2781] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:26.636937 16632 raft_consensus.cc:482] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:26.637058 16632 raft_consensus.cc:3037] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:26.639037 16632 raft_consensus.cc:504] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:26.639546 16632 leader_election.cc:288] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:26.640370 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa5d6450d9471395bda1af8c8c7973" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:26.640573 12859 raft_consensus.cc:3037] T 65fa5d6450d9471395bda1af8c8c7973 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:26.642431 12859 raft_consensus.cc:2454] T 65fa5d6450d9471395bda1af8c8c7973 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:26.642783 15819 leader_election.cc:302] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:26.642786 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fa5d6450d9471395bda1af8c8c7973" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:26.642967 12992 raft_consensus.cc:3037] T 65fa5d6450d9471395bda1af8c8c7973 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:26.643051 16865 raft_consensus.cc:2781] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:26.643476 16865 raft_consensus.cc:686] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:26.643759 16865 consensus_queue.cc:237] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:26.645718 12992 raft_consensus.cc:2454] T 65fa5d6450d9471395bda1af8c8c7973 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:26.648801 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:26.649371 12584 catalog_manager.cc:5484] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:26.656909 17318 tablet.cc:1722] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:26.657368 17318 tablet.cc:1620] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:26.659072 17319 tablet.cc:1722] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:26.659437 17319 tablet.cc:1620] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:26.668720 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=1bb3163584b0496499f130307583a3da] alter complete (version 2)
I20240502 08:18:26.677556 17320 tablet.cc:1722] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:26.678009 17320 tablet.cc:1620] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: MemRowSet was empty: no flush needed.
I20240502 08:18:26.773829 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:26.780441 17318 tablet.cc:1722] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:26.781365 17320 tablet.cc:1722] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:26.781538 17319 tablet.cc:1722] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:26.789662 12588 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=1bb3163584b0496499f130307583a3da] alter complete (version 3)
I20240502 08:18:26.891418 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:26.891642 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:26.895215 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet a0f8eb0abb574a3aa82848c1d60c77ab
I20240502 08:18:26.896107 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f8eb0abb574a3aa82848c1d60c77ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:26 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:26.896426 15599 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f8eb0abb574a3aa82848c1d60c77ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:26 UTC) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:26.896716 17325 tablet_replica.cc:330] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:26.897001 17325 raft_consensus.cc:2229] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:26.897248 17325 raft_consensus.cc:2258] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:26.898123 17325 ts_tablet_manager.cc:1875] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:26.898188 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet a0f8eb0abb574a3aa82848c1d60c77ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:26 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:26.905957 17325 ts_tablet_manager.cc:1888] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:26.905968 17312 consensus_queue.cc:1035] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:26.906124 17325 log.cc:1196] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/a0f8eb0abb574a3aa82848c1d60c77ab
I20240502 08:18:26.906632 17325 ts_tablet_manager.cc:1909] T a0f8eb0abb574a3aa82848c1d60c77ab P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:26.907829 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet a0f8eb0abb574a3aa82848c1d60c77ab (table rebalancer_dynamic_table_new_name [id=1bb3163584b0496499f130307583a3da]) successfully deleted
I20240502 08:18:26.910689 17327 tablet_replica.cc:330] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:26.910885 17327 raft_consensus.cc:2229] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:26.911243 17327 raft_consensus.cc:2258] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:26.911861 17326 tablet_replica.cc:330] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:26.911962 17327 ts_tablet_manager.cc:1875] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:26.912256 17326 raft_consensus.cc:2229] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:26.912523 17326 raft_consensus.cc:2258] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:26.913447 17326 ts_tablet_manager.cc:1875] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:26.916945 17327 ts_tablet_manager.cc:1888] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:26.917053 17327 log.cc:1196] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/a0f8eb0abb574a3aa82848c1d60c77ab
I20240502 08:18:26.917418 17327 ts_tablet_manager.cc:1909] T a0f8eb0abb574a3aa82848c1d60c77ab P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:26.919056 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet a0f8eb0abb574a3aa82848c1d60c77ab (table rebalancer_dynamic_table_new_name [id=1bb3163584b0496499f130307583a3da]) successfully deleted
I20240502 08:18:26.919417 17326 ts_tablet_manager.cc:1888] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:26.919598 17326 log.cc:1196] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/a0f8eb0abb574a3aa82848c1d60c77ab
I20240502 08:18:26.920043 17326 ts_tablet_manager.cc:1909] T a0f8eb0abb574a3aa82848c1d60c77ab P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting consensus metadata
I20240502 08:18:26.921118 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet a0f8eb0abb574a3aa82848c1d60c77ab (table rebalancer_dynamic_table_new_name [id=1bb3163584b0496499f130307583a3da]) successfully deleted
I20240502 08:18:26.927950 17331 consensus_queue.cc:1035] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:26.961285 17081 consensus_queue.cc:1035] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:26.972339 17315 consensus_queue.cc:1035] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:26.997627 12584 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:27.005398 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 8b98b9c9909245e6b416d002508322d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e659548c21e7476b8093e8ec130b771d]), partition=
I20240502 08:18:27.005537 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 8b98b9c9909245e6b416d002508322d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e659548c21e7476b8093e8ec130b771d]), partition=
I20240502 08:18:27.005779 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b98b9c9909245e6b416d002508322d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.005895 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b98b9c9909245e6b416d002508322d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.005939 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 8b98b9c9909245e6b416d002508322d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e659548c21e7476b8093e8ec130b771d]), partition=
I20240502 08:18:27.006239 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b98b9c9909245e6b416d002508322d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.010509 17337 tablet_bootstrap.cc:492] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:27.012262 17337 tablet_bootstrap.cc:654] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.023972 17340 tablet_bootstrap.cc:492] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:27.025534 17340 tablet_bootstrap.cc:654] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.025539 16865 consensus_queue.cc:1035] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:27.027246 17338 tablet_bootstrap.cc:492] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:27.029274 17338 tablet_bootstrap.cc:654] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.036150 16632 consensus_queue.cc:1035] T 65fa5d6450d9471395bda1af8c8c7973 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:27.039381 17338 tablet_bootstrap.cc:492] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:27.039889 17337 tablet_bootstrap.cc:492] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:27.040050 17337 ts_tablet_manager.cc:1367] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.015s
I20240502 08:18:27.040088 17338 ts_tablet_manager.cc:1367] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240502 08:18:27.040840 17337 raft_consensus.cc:348] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:27.041074 17337 raft_consensus.cc:374] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.041218 17337 raft_consensus.cc:729] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.041487 17337 consensus_queue.cc:260] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:27.042165 17337 ts_tablet_manager.cc:1398] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:27.040983 17338 raft_consensus.cc:348] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:27.042531 17338 raft_consensus.cc:374] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.044190 17338 raft_consensus.cc:729] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.044471 17338 consensus_queue.cc:260] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:27.045079 17338 ts_tablet_manager.cc:1398] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:27.057873 17340 tablet_bootstrap.cc:492] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:27.058020 17340 ts_tablet_manager.cc:1367] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.006s
I20240502 08:18:27.059023 17340 raft_consensus.cc:348] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:27.059235 17340 raft_consensus.cc:374] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.059311 17340 raft_consensus.cc:729] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.059525 17340 consensus_queue.cc:260] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:27.060027 17340 ts_tablet_manager.cc:1398] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:27.222059 17205 raft_consensus.cc:482] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:27.222254 17205 raft_consensus.cc:504] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:27.223049 17205 leader_election.cc:288] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:27.223827 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b98b9c9909245e6b416d002508322d1" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:27.223830 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b98b9c9909245e6b416d002508322d1" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:27.224102 12861 raft_consensus.cc:2454] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:27.224110 15486 raft_consensus.cc:2454] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:27.224532 13056 leader_election.cc:302] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:27.224846 17347 raft_consensus.cc:2781] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:27.224951 17347 raft_consensus.cc:482] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:27.225090 17347 raft_consensus.cc:3037] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.227433 17347 raft_consensus.cc:504] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:27.228039 17347 leader_election.cc:288] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:27.228361 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b98b9c9909245e6b416d002508322d1" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:27.228477 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b98b9c9909245e6b416d002508322d1" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:27.228559 12859 raft_consensus.cc:3037] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.228691 15486 raft_consensus.cc:3037] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.228710 12585 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:27.231283 15486 raft_consensus.cc:2454] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:27.231282 12859 raft_consensus.cc:2454] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:27.231724 13056 leader_election.cc:302] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:27.231983 17347 raft_consensus.cc:2781] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:27.232679 17347 raft_consensus.cc:686] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:18:27.232962 17347 consensus_queue.cc:237] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:27.239457 15599 tablet_service.cc:1467] Processing CreateTablet for tablet f3f9adebeded4492a2b18b9b60de1c79 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b1448b5e49614c108f8d76ba41e5da72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:27.239562 12585 catalog_manager.cc:5484] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:27.239873 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f9adebeded4492a2b18b9b60de1c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.242877 15467 tablet_service.cc:1467] Processing CreateTablet for tablet f3f9adebeded4492a2b18b9b60de1c79 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b1448b5e49614c108f8d76ba41e5da72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:27.243266 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f9adebeded4492a2b18b9b60de1c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.243330 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 087809a059d14b1f9a24ec3e956e1305 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b1448b5e49614c108f8d76ba41e5da72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:27.243678 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087809a059d14b1f9a24ec3e956e1305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.244215 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 087809a059d14b1f9a24ec3e956e1305 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b1448b5e49614c108f8d76ba41e5da72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:27.244567 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087809a059d14b1f9a24ec3e956e1305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.244721 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 087809a059d14b1f9a24ec3e956e1305 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b1448b5e49614c108f8d76ba41e5da72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:27.245112 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087809a059d14b1f9a24ec3e956e1305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.247342 12710 tablet_service.cc:1467] Processing CreateTablet for tablet f3f9adebeded4492a2b18b9b60de1c79 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b1448b5e49614c108f8d76ba41e5da72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:27.247721 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f9adebeded4492a2b18b9b60de1c79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.251240 12972 tablet_service.cc:1467] Processing CreateTablet for tablet df8d4e22f51b4493a0ca437352c846e7 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b1448b5e49614c108f8d76ba41e5da72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:27.251454 15468 tablet_service.cc:1467] Processing CreateTablet for tablet df8d4e22f51b4493a0ca437352c846e7 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b1448b5e49614c108f8d76ba41e5da72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:27.251627 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8d4e22f51b4493a0ca437352c846e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.251804 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8d4e22f51b4493a0ca437352c846e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.253508 15733 tablet_service.cc:1467] Processing CreateTablet for tablet df8d4e22f51b4493a0ca437352c846e7 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b1448b5e49614c108f8d76ba41e5da72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:27.253911 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8d4e22f51b4493a0ca437352c846e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.272934 17338 tablet_bootstrap.cc:492] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:27.274432 17338 tablet_bootstrap.cc:654] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.281875 17340 tablet_bootstrap.cc:492] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:27.288098 17340 tablet_bootstrap.cc:654] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.292779 17354 tablet_bootstrap.cc:492] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:27.296417 17354 tablet_bootstrap.cc:654] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.299341 17340 tablet_bootstrap.cc:492] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:27.300354 17338 tablet_bootstrap.cc:492] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:27.300616 17338 ts_tablet_manager.cc:1367] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.006s
I20240502 08:18:27.301486 17338 raft_consensus.cc:348] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:27.301767 17338 raft_consensus.cc:374] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.301870 17338 raft_consensus.cc:729] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.302100 17338 consensus_queue.cc:260] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:27.302759 17352 tablet_bootstrap.cc:492] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:27.304093 17350 tablet_bootstrap.cc:492] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:27.305642 17340 ts_tablet_manager.cc:1367] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.005s
I20240502 08:18:27.306218 17338 ts_tablet_manager.cc:1398] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:18:27.307003 17340 raft_consensus.cc:348] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:27.308768 17352 tablet_bootstrap.cc:654] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.309094 17338 tablet_bootstrap.cc:492] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:27.310712 17338 tablet_bootstrap.cc:654] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.310858 17340 raft_consensus.cc:374] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.310985 17340 raft_consensus.cc:729] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.311261 17340 consensus_queue.cc:260] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:27.311790 17340 ts_tablet_manager.cc:1398] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 08:18:27.312893 17350 tablet_bootstrap.cc:654] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.317139 17352 tablet_bootstrap.cc:492] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:27.317272 17352 ts_tablet_manager.cc:1367] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240502 08:18:27.316340 17349 tablet_bootstrap.cc:492] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:27.318040 17352 raft_consensus.cc:348] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.319281 17349 tablet_bootstrap.cc:654] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.321020 17338 tablet_bootstrap.cc:492] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:27.321202 17338 ts_tablet_manager.cc:1367] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 08:18:27.321926 17338 raft_consensus.cc:348] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.322182 17338 raft_consensus.cc:374] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.322299 17338 raft_consensus.cc:729] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.322554 17338 consensus_queue.cc:260] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.323201 17338 ts_tablet_manager.cc:1398] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:27.325336 17227 raft_consensus.cc:482] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:27.325505 17227 raft_consensus.cc:504] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.326117 17227 leader_election.cc:288] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:27.326857 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f9adebeded4492a2b18b9b60de1c79" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
W20240502 08:18:27.328317 15421 leader_election.cc:341] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:27.329051 17354 tablet_bootstrap.cc:492] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:27.329206 17354 ts_tablet_manager.cc:1367] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.007s
I20240502 08:18:27.329981 17354 raft_consensus.cc:348] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:27.330165 17354 raft_consensus.cc:374] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.330252 17354 raft_consensus.cc:729] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.327133 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f9adebeded4492a2b18b9b60de1c79" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:27.331710 17352 raft_consensus.cc:374] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.331822 17352 raft_consensus.cc:729] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.332046 17352 consensus_queue.cc:260] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.332327 12730 raft_consensus.cc:2454] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:27.333379 15419 leader_election.cc:302] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:27.333484 17350 tablet_bootstrap.cc:492] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:27.333619 17350 ts_tablet_manager.cc:1367] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.003s
I20240502 08:18:27.333881 17227 raft_consensus.cc:2781] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:27.333947 17349 tablet_bootstrap.cc:492] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:27.333974 17227 raft_consensus.cc:482] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:27.334074 17227 raft_consensus.cc:3037] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.334079 17349 ts_tablet_manager.cc:1367] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20240502 08:18:27.334427 17350 raft_consensus.cc:348] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:27.334651 17350 raft_consensus.cc:374] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.334609 17354 consensus_queue.cc:260] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:27.334738 17350 raft_consensus.cc:729] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.334826 17349 raft_consensus.cc:348] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:27.334995 17349 raft_consensus.cc:374] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.335073 17349 raft_consensus.cc:729] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.335376 17352 ts_tablet_manager.cc:1398] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240502 08:18:27.335301 17349 consensus_queue.cc:260] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:27.336205 17350 consensus_queue.cc:260] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:27.336710 17227 raft_consensus.cc:504] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.336937 17349 ts_tablet_manager.cc:1398] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:27.337215 17349 tablet_bootstrap.cc:492] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:27.338128 17354 ts_tablet_manager.cc:1398] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 08:18:27.338799 17349 tablet_bootstrap.cc:654] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.340238 17350 ts_tablet_manager.cc:1398] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 08:18:27.340606 17350 tablet_bootstrap.cc:492] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:27.344456 17350 tablet_bootstrap.cc:654] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.346325 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f9adebeded4492a2b18b9b60de1c79" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
W20240502 08:18:27.347010 15421 leader_election.cc:341] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:27.347677 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f9adebeded4492a2b18b9b60de1c79" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:27.347900 12730 raft_consensus.cc:3037] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.348982 17349 tablet_bootstrap.cc:492] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:27.349155 17349 ts_tablet_manager.cc:1367] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20240502 08:18:27.349438 17227 leader_election.cc:288] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:27.349920 17349 raft_consensus.cc:348] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.350107 17349 raft_consensus.cc:374] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.350171 17349 raft_consensus.cc:729] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.350404 17349 consensus_queue.cc:260] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.350620 12730 raft_consensus.cc:2454] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:27.351475 15419 leader_election.cc:302] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:27.354148 17227 raft_consensus.cc:2781] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:27.354233 17349 ts_tablet_manager.cc:1398] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:27.355260 17227 raft_consensus.cc:686] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:27.355580 17227 consensus_queue.cc:237] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.359885 17350 tablet_bootstrap.cc:492] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:27.360143 17350 ts_tablet_manager.cc:1367] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20240502 08:18:27.361107 17350 raft_consensus.cc:348] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:27.361337 17350 raft_consensus.cc:374] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.361418 17350 raft_consensus.cc:729] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.361670 17350 consensus_queue.cc:260] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:27.362385 17350 ts_tablet_manager.cc:1398] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:27.373214 12585 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:27.379519 15486 raft_consensus.cc:1261] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:27.380043 12859 raft_consensus.cc:1261] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:27.380053 17347 consensus_queue.cc:1035] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:27.380674 17263 raft_consensus.cc:482] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:27.380848 17263 raft_consensus.cc:504] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.381669 17263 leader_election.cc:288] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:27.381807 17347 consensus_queue.cc:1035] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:27.382834 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f9adebeded4492a2b18b9b60de1c79" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:27.383132 12730 raft_consensus.cc:2379] T f3f9adebeded4492a2b18b9b60de1c79 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in current term 1: Already voted for candidate db7893bb29a047b3b82cc60bfd8930a3 in this term.
I20240502 08:18:27.383248 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f9adebeded4492a2b18b9b60de1c79" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:27.384115 15554 leader_election.cc:302] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:27.384663 17263 raft_consensus.cc:3037] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.389395 17263 raft_consensus.cc:2726] T f3f9adebeded4492a2b18b9b60de1c79 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:27.391745 12585 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet 8b98b9c9909245e6b416d002508322d1 (table rebalancer_dynamic_table [id=e659548c21e7476b8093e8ec130b771d]). Expected version 1 got 0
I20240502 08:18:27.400272 17366 tablet.cc:1722] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:27.401127 17365 tablet.cc:1722] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:27.401916 17365 tablet.cc:1620] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:27.402565 17366 tablet.cc:1620] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:27.405143 12585 catalog_manager.cc:5484] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:27.412384 12585 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet 8b98b9c9909245e6b416d002508322d1 (table rebalancer_dynamic_table [id=e659548c21e7476b8093e8ec130b771d]). Expected version 1 got 0
I20240502 08:18:27.416826 17367 tablet.cc:1722] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:27.417817 17367 tablet.cc:1620] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:27.419167 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet 8b98b9c9909245e6b416d002508322d1 (table rebalancer_dynamic_table [id=e659548c21e7476b8093e8ec130b771d]). Expected version 1 got 0
I20240502 08:18:27.419396 12588 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=e659548c21e7476b8093e8ec130b771d] alter complete (version 1)
I20240502 08:18:27.433246 17227 raft_consensus.cc:482] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:27.433414 17227 raft_consensus.cc:504] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:27.434273 17227 leader_election.cc:288] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:27.434739 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8d4e22f51b4493a0ca437352c846e7" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:27.435021 12992 raft_consensus.cc:2454] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:27.434973 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8d4e22f51b4493a0ca437352c846e7" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:27.435192 15754 raft_consensus.cc:2454] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:27.435621 15419 leader_election.cc:302] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:27.435900 17227 raft_consensus.cc:2781] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:27.436020 17227 raft_consensus.cc:482] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:27.436107 17227 raft_consensus.cc:3037] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.438169 17227 raft_consensus.cc:504] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:27.438797 17227 leader_election.cc:288] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:27.439412 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8d4e22f51b4493a0ca437352c846e7" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:27.439466 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8d4e22f51b4493a0ca437352c846e7" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:27.439610 12992 raft_consensus.cc:3037] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.439646 15755 raft_consensus.cc:3037] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.442025 12992 raft_consensus.cc:2454] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:27.442238 15755 raft_consensus.cc:2454] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:27.442425 15419 leader_election.cc:302] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:27.442721 17227 raft_consensus.cc:2781] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:27.442919 17227 raft_consensus.cc:686] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:27.443230 17227 consensus_queue.cc:237] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:27.446956 12584 catalog_manager.cc:5484] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:27.526531 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:27.535723 17367 tablet.cc:1722] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:27.535764 17365 tablet.cc:1722] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:27.536247 17365 tablet.cc:1620] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:27.536420 17367 tablet.cc:1620] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:27.537781 17366 tablet.cc:1722] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:27.538267 17366 tablet.cc:1620] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:27.546576 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=e659548c21e7476b8093e8ec130b771d] alter complete (version 2)
I20240502 08:18:27.568578 17347 raft_consensus.cc:482] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:27.568753 17347 raft_consensus.cc:504] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:27.569459 17347 leader_election.cc:288] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:27.569826 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087809a059d14b1f9a24ec3e956e1305" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:27.569902 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087809a059d14b1f9a24ec3e956e1305" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:27.570046 12992 raft_consensus.cc:2454] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:27.570132 15621 raft_consensus.cc:2454] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:27.570505 13058 leader_election.cc:302] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:27.570789 17347 raft_consensus.cc:2781] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:27.570907 17347 raft_consensus.cc:482] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:27.570995 17347 raft_consensus.cc:3037] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.573182 17347 raft_consensus.cc:504] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:27.573767 17347 leader_election.cc:288] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:27.574343 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087809a059d14b1f9a24ec3e956e1305" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:27.574357 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087809a059d14b1f9a24ec3e956e1305" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:27.574549 15621 raft_consensus.cc:3037] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.574545 12992 raft_consensus.cc:3037] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.576479 15621 raft_consensus.cc:2454] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:27.576504 12992 raft_consensus.cc:2454] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:27.576994 13056 leader_election.cc:302] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:27.577461 17347 raft_consensus.cc:2781] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:27.577631 17347 raft_consensus.cc:686] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:18:27.577904 17347 consensus_queue.cc:237] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:27.581708 12585 catalog_manager.cc:5484] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:27.649286 12585 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:27.664611 17365 tablet.cc:1722] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:27.667438 17367 tablet.cc:1722] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:27.671383 17366 tablet.cc:1722] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:27.676430 12585 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=e659548c21e7476b8093e8ec130b771d] alter complete (version 3)
I20240502 08:18:27.777858 12585 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:27.778139 12585 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:27.781047 12585 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 8b98b9c9909245e6b416d002508322d1
I20240502 08:18:27.781744 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b98b9c9909245e6b416d002508322d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:27 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:27.782191 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b98b9c9909245e6b416d002508322d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:27 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:27.782647 17371 tablet_replica.cc:330] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:27.782619 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b98b9c9909245e6b416d002508322d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:27 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:27.782878 17371 raft_consensus.cc:2229] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:27.783103 17371 raft_consensus.cc:2258] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:27.783262 17373 tablet_replica.cc:330] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:27.783470 17373 raft_consensus.cc:2229] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:27.783732 17373 raft_consensus.cc:2258] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:27.784013 17371 ts_tablet_manager.cc:1875] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:27.784547 17373 ts_tablet_manager.cc:1875] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:27.790603 17373 ts_tablet_manager.cc:1888] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:27.790792 17373 log.cc:1196] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/8b98b9c9909245e6b416d002508322d1
I20240502 08:18:27.791239 17373 ts_tablet_manager.cc:1909] T 8b98b9c9909245e6b416d002508322d1 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:27.792248 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 8b98b9c9909245e6b416d002508322d1 (table rebalancer_dynamic_table_new_name [id=e659548c21e7476b8093e8ec130b771d]) successfully deleted
I20240502 08:18:27.792459 17371 ts_tablet_manager.cc:1888] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:27.792573 17371 log.cc:1196] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/8b98b9c9909245e6b416d002508322d1
I20240502 08:18:27.792963 17371 ts_tablet_manager.cc:1909] T 8b98b9c9909245e6b416d002508322d1 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:27.793650 17372 tablet_replica.cc:330] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:27.793773 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 8b98b9c9909245e6b416d002508322d1 (table rebalancer_dynamic_table_new_name [id=e659548c21e7476b8093e8ec130b771d]) successfully deleted
I20240502 08:18:27.793843 17372 raft_consensus.cc:2229] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:27.794175 17372 raft_consensus.cc:2258] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:27.794824 17372 ts_tablet_manager.cc:1875] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:27.799445 17372 ts_tablet_manager.cc:1888] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:27.799585 17372 log.cc:1196] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/8b98b9c9909245e6b416d002508322d1
I20240502 08:18:27.800015 17372 ts_tablet_manager.cc:1909] T 8b98b9c9909245e6b416d002508322d1 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:27.800891 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 8b98b9c9909245e6b416d002508322d1 (table rebalancer_dynamic_table_new_name [id=e659548c21e7476b8093e8ec130b771d]) successfully deleted
I20240502 08:18:27.826573 17227 consensus_queue.cc:1035] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:27.830757 17315 consensus_queue.cc:1035] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:27.836099 17315 consensus_queue.cc:1035] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:27.841928 17227 consensus_queue.cc:1035] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:27.883623 12588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:27.891126 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 2847555c923647dd979c4ecc5691a3cb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=473385eff87646438963eeeb3608c873]), partition=
I20240502 08:18:27.891285 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 2847555c923647dd979c4ecc5691a3cb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=473385eff87646438963eeeb3608c873]), partition=
I20240502 08:18:27.891491 12710 tablet_service.cc:1467] Processing CreateTablet for tablet 2847555c923647dd979c4ecc5691a3cb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=473385eff87646438963eeeb3608c873]), partition=
I20240502 08:18:27.891563 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2847555c923647dd979c4ecc5691a3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.891594 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2847555c923647dd979c4ecc5691a3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.891861 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2847555c923647dd979c4ecc5691a3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:27.898779 17384 tablet_bootstrap.cc:492] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:27.903028 17384 tablet_bootstrap.cc:654] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.905077 17384 tablet_bootstrap.cc:492] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:27.905289 17384 ts_tablet_manager.cc:1367] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 08:18:27.906013 17384 raft_consensus.cc:348] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.906179 17384 raft_consensus.cc:374] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.906278 17384 raft_consensus.cc:729] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.906518 17384 consensus_queue.cc:260] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.907264 17384 ts_tablet_manager.cc:1398] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:27.908807 17383 tablet_bootstrap.cc:492] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:27.910012 17385 tablet_bootstrap.cc:492] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:27.910714 17383 tablet_bootstrap.cc:654] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.911825 17385 tablet_bootstrap.cc:654] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:27.912349 17383 tablet_bootstrap.cc:492] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:27.912482 17383 ts_tablet_manager.cc:1367] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:27.913362 17383 raft_consensus.cc:348] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.913648 17383 raft_consensus.cc:374] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.913806 17383 raft_consensus.cc:729] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.914108 17383 consensus_queue.cc:260] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.917873 17385 tablet_bootstrap.cc:492] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:27.918195 17385 ts_tablet_manager.cc:1367] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20240502 08:18:27.917904 17383 ts_tablet_manager.cc:1398] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:27.918989 17385 raft_consensus.cc:348] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.919183 17385 raft_consensus.cc:374] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:27.919306 17385 raft_consensus.cc:729] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:27.919510 17385 consensus_queue.cc:260] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.919966 17385 ts_tablet_manager.cc:1398] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:27.937630 17331 raft_consensus.cc:482] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:27.937855 17331 raft_consensus.cc:504] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.938639 17331 leader_election.cc:288] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:27.939033 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2847555c923647dd979c4ecc5691a3cb" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:27.939159 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2847555c923647dd979c4ecc5691a3cb" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:27.939312 15886 raft_consensus.cc:2454] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:27.939419 12730 raft_consensus.cc:2454] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:27.939796 12797 leader_election.cc:302] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:27.940377 17331 raft_consensus.cc:2781] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:27.940487 17331 raft_consensus.cc:482] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:27.944455 17331 raft_consensus.cc:3037] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.946940 17331 raft_consensus.cc:504] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.947669 17331 leader_election.cc:288] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:27.948011 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2847555c923647dd979c4ecc5691a3cb" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:27.948240 15886 raft_consensus.cc:3037] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.950193 15886 raft_consensus.cc:2454] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:27.950539 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2847555c923647dd979c4ecc5691a3cb" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:27.950572 12797 leader_election.cc:302] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:27.950740 12730 raft_consensus.cc:3037] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:27.950839 17389 raft_consensus.cc:2781] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:27.950990 17389 raft_consensus.cc:686] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:18:27.951319 17389 consensus_queue.cc:237] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:27.953269 12730 raft_consensus.cc:2454] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:27.955638 12588 catalog_manager.cc:5484] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:27.999552 17369 consensus_queue.cc:1035] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:28.003917 17347 consensus_queue.cc:1035] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:28.069227 12588 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:28.075235 12727 raft_consensus.cc:1261] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:28.075544 15886 raft_consensus.cc:1261] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:28.076071 17331 consensus_queue.cc:1035] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:28.076426 17389 consensus_queue.cc:1035] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:28.079804 17366 tablet.cc:1722] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:28.080243 17366 tablet.cc:1620] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:28.081161 12585 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet 2847555c923647dd979c4ecc5691a3cb (table rebalancer_dynamic_table [id=473385eff87646438963eeeb3608c873]). Expected version 1 got 0
I20240502 08:18:28.082779 17395 tablet.cc:1722] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:28.083524 17395 tablet.cc:1620] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:28.089304 12588 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=473385eff87646438963eeeb3608c873] alter complete (version 1)
I20240502 08:18:28.094332 12588 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 2847555c923647dd979c4ecc5691a3cb (table rebalancer_dynamic_table [id=473385eff87646438963eeeb3608c873]). Expected version 1 got 0
I20240502 08:18:28.096917 17397 tablet.cc:1722] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:28.097579 17397 tablet.cc:1620] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:28.098616 12588 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 2847555c923647dd979c4ecc5691a3cb (table rebalancer_dynamic_table [id=473385eff87646438963eeeb3608c873]). Expected version 1 got 0
I20240502 08:18:28.150772 12588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:28.160102 13101 tablet_service.cc:1467] Processing CreateTablet for tablet a5af24c28aa04f72844744771f6b7249 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ac4fff256cff4b5fbc1f66de9625967c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:28.160595 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5af24c28aa04f72844744771f6b7249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.161166 12970 tablet_service.cc:1467] Processing CreateTablet for tablet e0b58d36dccb4112a6fcb603964e91b3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ac4fff256cff4b5fbc1f66de9625967c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:28.161556 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b58d36dccb4112a6fcb603964e91b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.161557 15599 tablet_service.cc:1467] Processing CreateTablet for tablet e0b58d36dccb4112a6fcb603964e91b3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ac4fff256cff4b5fbc1f66de9625967c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:28.161999 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b58d36dccb4112a6fcb603964e91b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.161988 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 6546b77d0fd3434487dfe3abd335803e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ac4fff256cff4b5fbc1f66de9625967c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:28.162369 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6546b77d0fd3434487dfe3abd335803e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.162825 15601 tablet_service.cc:1467] Processing CreateTablet for tablet a5af24c28aa04f72844744771f6b7249 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ac4fff256cff4b5fbc1f66de9625967c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:28.163190 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5af24c28aa04f72844744771f6b7249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.165675 12841 tablet_service.cc:1467] Processing CreateTablet for tablet a5af24c28aa04f72844744771f6b7249 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ac4fff256cff4b5fbc1f66de9625967c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:28.166045 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5af24c28aa04f72844744771f6b7249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.169234 17385 tablet_bootstrap.cc:492] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:28.170879 17385 tablet_bootstrap.cc:654] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:28.170827 15865 tablet_service.cc:1467] Processing CreateTablet for tablet e0b58d36dccb4112a6fcb603964e91b3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ac4fff256cff4b5fbc1f66de9625967c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:28.171223 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b58d36dccb4112a6fcb603964e91b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.179216 17383 tablet_bootstrap.cc:492] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:28.180101 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 6546b77d0fd3434487dfe3abd335803e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ac4fff256cff4b5fbc1f66de9625967c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:28.180537 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6546b77d0fd3434487dfe3abd335803e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.181376 17383 tablet_bootstrap.cc:654] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:28.182847 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 6546b77d0fd3434487dfe3abd335803e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=ac4fff256cff4b5fbc1f66de9625967c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:28.183241 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6546b77d0fd3434487dfe3abd335803e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.190773 17385 tablet_bootstrap.cc:492] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:28.191080 17385 ts_tablet_manager.cc:1367] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.007s
I20240502 08:18:28.192071 17385 raft_consensus.cc:348] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:28.193886 17385 raft_consensus.cc:374] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:28.194238 17385 raft_consensus.cc:729] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:28.194922 12585 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:28.194926 17383 tablet_bootstrap.cc:492] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:28.195400 17383 ts_tablet_manager.cc:1367] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.001s
I20240502 08:18:28.196215 17383 raft_consensus.cc:348] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.197402 17383 raft_consensus.cc:374] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:28.197549 17383 raft_consensus.cc:729] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:28.197845 17383 consensus_queue.cc:260] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.198647 17383 ts_tablet_manager.cc:1398] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:28.198987 17385 consensus_queue.cc:260] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:28.200841 17385 ts_tablet_manager.cc:1398] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 08:18:28.201185 17385 tablet_bootstrap.cc:492] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:28.202962 17385 tablet_bootstrap.cc:654] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:28.213887 17366 tablet.cc:1722] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:28.214427 17366 tablet.cc:1620] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:28.226634 17395 tablet.cc:1722] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:28.227216 17395 tablet.cc:1620] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:28.228339 12588 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=473385eff87646438963eeeb3608c873] alter complete (version 2)
I20240502 08:18:28.232892 17404 tablet_bootstrap.cc:492] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:28.234654 17404 tablet_bootstrap.cc:654] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:28.234647 17397 tablet.cc:1722] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:28.235515 17397 tablet.cc:1620] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:28.231627 17403 tablet_bootstrap.cc:492] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:28.241191 17403 tablet_bootstrap.cc:654] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:28.245122 17401 tablet_bootstrap.cc:492] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:28.251848 17404 tablet_bootstrap.cc:492] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:28.252045 17404 ts_tablet_manager.cc:1367] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240502 08:18:28.252982 17404 raft_consensus.cc:348] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:28.253592 17401 tablet_bootstrap.cc:654] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:28.254510 17404 raft_consensus.cc:374] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:28.254664 17404 raft_consensus.cc:729] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:28.255378 17404 consensus_queue.cc:260] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:28.256337 17404 ts_tablet_manager.cc:1398] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:28.257336 17403 tablet_bootstrap.cc:492] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:28.258510 17403 ts_tablet_manager.cc:1367] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
W20240502 08:18:28.261046 12666 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.6:44983, user_credentials={real_user=slave}} blocked reactor thread for 50804.6us
I20240502 08:18:28.261003 17403 raft_consensus.cc:348] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:28.261368 17403 raft_consensus.cc:374] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:28.261507 17403 raft_consensus.cc:729] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:28.261799 17403 consensus_queue.cc:260] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:28.262566 17403 ts_tablet_manager.cc:1398] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:28.263047 17403 tablet_bootstrap.cc:492] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:28.265237 17400 tablet_bootstrap.cc:492] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:28.267728 17403 tablet_bootstrap.cc:654] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:28.270265 16632 raft_consensus.cc:482] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:28.270435 16632 raft_consensus.cc:504] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.271157 16632 leader_election.cc:288] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:28.271701 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b58d36dccb4112a6fcb603964e91b3" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
W20240502 08:18:28.272202 15819 leader_election.cc:341] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:28.272590 17400 tablet_bootstrap.cc:654] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:28.274236 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b58d36dccb4112a6fcb603964e91b3" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
W20240502 08:18:28.275067 15822 leader_election.cc:341] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:28.275199 15822 leader_election.cc:302] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10f0e36203914b85b77e78609ffced98; no voters: 60b6e265e7754b7187307d05ef405dd4, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:28.275946 16632 raft_consensus.cc:2726] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:28.282454 17400 tablet_bootstrap.cc:492] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:28.282667 17400 ts_tablet_manager.cc:1367] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 08:18:28.283896 17400 raft_consensus.cc:348] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.284137 17400 raft_consensus.cc:374] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:28.284291 17400 raft_consensus.cc:729] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:28.284570 17400 consensus_queue.cc:260] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.285213 17401 tablet_bootstrap.cc:492] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:28.287068 17400 ts_tablet_manager.cc:1398] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:28.287434 17401 ts_tablet_manager.cc:1367] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.004s
I20240502 08:18:28.288189 17401 raft_consensus.cc:348] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.291916 17401 raft_consensus.cc:374] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:28.292069 17401 raft_consensus.cc:729] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:28.292454 17385 tablet_bootstrap.cc:492] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:28.292456 17401 consensus_queue.cc:260] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.292604 17385 ts_tablet_manager.cc:1367] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.091s user 0.019s sys 0.007s
I20240502 08:18:28.293408 17385 raft_consensus.cc:348] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.293602 17385 raft_consensus.cc:374] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:28.293690 17385 raft_consensus.cc:729] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:28.293905 17385 consensus_queue.cc:260] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.294488 17385 ts_tablet_manager.cc:1398] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:28.297243 17401 ts_tablet_manager.cc:1398] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 08:18:28.302732 17401 tablet_bootstrap.cc:492] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:28.306212 17403 tablet_bootstrap.cc:492] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:28.306381 17403 ts_tablet_manager.cc:1367] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20240502 08:18:28.307315 17401 tablet_bootstrap.cc:654] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:28.308133 17403 raft_consensus.cc:348] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.308352 17403 raft_consensus.cc:374] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:28.308427 17403 raft_consensus.cc:729] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:28.308678 17403 consensus_queue.cc:260] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.309221 17403 ts_tablet_manager.cc:1398] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:28.312812 17401 tablet_bootstrap.cc:492] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:28.313017 17401 ts_tablet_manager.cc:1367] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240502 08:18:28.315569 17401 raft_consensus.cc:348] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.315832 17401 raft_consensus.cc:374] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:28.315928 17401 raft_consensus.cc:729] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:28.316246 17401 consensus_queue.cc:260] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.316881 17401 ts_tablet_manager.cc:1398] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:28.340440 12585 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:28.347911 17366 tablet.cc:1722] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:28.348724 17395 tablet.cc:1722] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:28.360741 17397 tablet.cc:1722] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:28.363606 12585 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=473385eff87646438963eeeb3608c873] alter complete (version 3)
I20240502 08:18:28.387532 17389 raft_consensus.cc:482] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:28.387674 17389 raft_consensus.cc:504] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:28.388351 17389 leader_election.cc:288] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:28.388693 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6546b77d0fd3434487dfe3abd335803e" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:28.388744 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6546b77d0fd3434487dfe3abd335803e" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:28.388882 12992 raft_consensus.cc:2454] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:28.388937 15754 raft_consensus.cc:2454] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:28.389250 12796 leader_election.cc:302] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:28.389513 17389 raft_consensus.cc:2781] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:28.389628 17389 raft_consensus.cc:482] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:28.389734 17389 raft_consensus.cc:3037] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:28.391707 17389 raft_consensus.cc:504] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:28.392252 17389 leader_election.cc:288] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:28.392761 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6546b77d0fd3434487dfe3abd335803e" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:28.392769 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6546b77d0fd3434487dfe3abd335803e" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:28.392967 12992 raft_consensus.cc:3037] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:28.393039 15754 raft_consensus.cc:3037] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:28.395069 12992 raft_consensus.cc:2454] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:28.395179 15754 raft_consensus.cc:2454] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:28.395412 12796 leader_election.cc:302] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:28.395685 17389 raft_consensus.cc:2781] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:28.395859 17389 raft_consensus.cc:686] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:18:28.396139 17389 consensus_queue.cc:237] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:28.399695 12585 catalog_manager.cc:5484] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:28.451459 17398 raft_consensus.cc:482] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:28.451624 17398 raft_consensus.cc:504] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.452594 17398 leader_election.cc:288] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:28.453155 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5af24c28aa04f72844744771f6b7249" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:28.453145 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5af24c28aa04f72844744771f6b7249" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:28.453372 12861 raft_consensus.cc:2454] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:28.453377 13119 raft_consensus.cc:2454] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:28.453792 15554 leader_election.cc:302] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:28.454047 17398 raft_consensus.cc:2781] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:28.454185 17398 raft_consensus.cc:482] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:28.454270 17398 raft_consensus.cc:3037] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:28.457171 17398 raft_consensus.cc:504] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.457922 17398 leader_election.cc:288] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:28.458281 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5af24c28aa04f72844744771f6b7249" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:28.458518 13119 raft_consensus.cc:3037] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:28.458439 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5af24c28aa04f72844744771f6b7249" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:28.458649 12861 raft_consensus.cc:3037] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:28.461342 13119 raft_consensus.cc:2454] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:28.461328 12861 raft_consensus.cc:2454] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:28.461727 15554 leader_election.cc:302] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:28.462476 17263 raft_consensus.cc:2781] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:28.462620 17263 raft_consensus.cc:686] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:28.462919 17263 consensus_queue.cc:237] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.469259 12585 catalog_manager.cc:5484] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:28.469388 12588 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:28.469614 12588 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:28.473116 12588 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 2847555c923647dd979c4ecc5691a3cb
I20240502 08:18:28.473898 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 2847555c923647dd979c4ecc5691a3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:28 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:28.474171 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 2847555c923647dd979c4ecc5691a3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:28 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:28.474639 12710 tablet_service.cc:1514] Processing DeleteTablet for tablet 2847555c923647dd979c4ecc5691a3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:28 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:28.474778 17420 tablet_replica.cc:330] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:28.474987 17420 raft_consensus.cc:2229] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:28.475234 17421 tablet_replica.cc:330] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:28.475224 17420 raft_consensus.cc:2258] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:28.475549 17421 raft_consensus.cc:2229] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:28.475816 17421 raft_consensus.cc:2258] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:28.476150 17420 ts_tablet_manager.cc:1875] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:28.477833 17421 ts_tablet_manager.cc:1875] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:28.484181 17421 ts_tablet_manager.cc:1888] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:28.484248 17420 ts_tablet_manager.cc:1888] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:28.484325 17421 log.cc:1196] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/2847555c923647dd979c4ecc5691a3cb
I20240502 08:18:28.484356 17420 log.cc:1196] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/2847555c923647dd979c4ecc5691a3cb
I20240502 08:18:28.484818 17421 ts_tablet_manager.cc:1909] T 2847555c923647dd979c4ecc5691a3cb P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:28.484872 17420 ts_tablet_manager.cc:1909] T 2847555c923647dd979c4ecc5691a3cb P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:28.485947 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 2847555c923647dd979c4ecc5691a3cb (table rebalancer_dynamic_table_new_name [id=473385eff87646438963eeeb3608c873]) successfully deleted
I20240502 08:18:28.486178 17419 tablet_replica.cc:330] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:28.486292 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 2847555c923647dd979c4ecc5691a3cb (table rebalancer_dynamic_table_new_name [id=473385eff87646438963eeeb3608c873]) successfully deleted
I20240502 08:18:28.486414 17419 raft_consensus.cc:2229] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:28.486840 17419 raft_consensus.cc:2258] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:28.487608 17419 ts_tablet_manager.cc:1875] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:28.492280 17419 ts_tablet_manager.cc:1888] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:28.492410 17419 log.cc:1196] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/2847555c923647dd979c4ecc5691a3cb
I20240502 08:18:28.492985 17419 ts_tablet_manager.cc:1909] T 2847555c923647dd979c4ecc5691a3cb P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:28.494187 17346 raft_consensus.cc:482] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:28.494221 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 2847555c923647dd979c4ecc5691a3cb (table rebalancer_dynamic_table_new_name [id=473385eff87646438963eeeb3608c873]) successfully deleted
I20240502 08:18:28.494364 17346 raft_consensus.cc:504] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.495371 17346 leader_election.cc:288] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:28.495748 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b58d36dccb4112a6fcb603964e91b3" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:28.495949 15887 raft_consensus.cc:2454] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:28.495995 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b58d36dccb4112a6fcb603964e91b3" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:28.496203 15620 raft_consensus.cc:2454] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:28.496408 12927 leader_election.cc:302] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:18:28.496670 17346 raft_consensus.cc:2781] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:28.496784 17346 raft_consensus.cc:482] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:28.496882 17346 raft_consensus.cc:3037] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:28.498857 17346 raft_consensus.cc:504] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.499526 17346 leader_election.cc:288] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:28.499902 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b58d36dccb4112a6fcb603964e91b3" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:28.499899 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0b58d36dccb4112a6fcb603964e91b3" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:28.500054 15887 raft_consensus.cc:3037] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:28.500136 15620 raft_consensus.cc:3037] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:28.502146 15620 raft_consensus.cc:2454] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:28.502215 15887 raft_consensus.cc:2454] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:28.502629 12927 leader_election.cc:302] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:18:28.502913 17346 raft_consensus.cc:2781] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:28.503338 17346 raft_consensus.cc:686] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:28.503643 17346 consensus_queue.cc:237] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:28.507845 12588 catalog_manager.cc:5484] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:28.575742 12588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:28.583480 12710 tablet_service.cc:1467] Processing CreateTablet for tablet f582c3bcc16141e2afbf2d0cb6691f1d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6cacde43f7df4d8591aacc49979244a0]), partition=
I20240502 08:18:28.583644 15733 tablet_service.cc:1467] Processing CreateTablet for tablet f582c3bcc16141e2afbf2d0cb6691f1d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6cacde43f7df4d8591aacc49979244a0]), partition=
I20240502 08:18:28.583635 15467 tablet_service.cc:1467] Processing CreateTablet for tablet f582c3bcc16141e2afbf2d0cb6691f1d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6cacde43f7df4d8591aacc49979244a0]), partition=
I20240502 08:18:28.583871 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f582c3bcc16141e2afbf2d0cb6691f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.583984 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f582c3bcc16141e2afbf2d0cb6691f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.583985 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f582c3bcc16141e2afbf2d0cb6691f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:28.588505 17424 tablet_bootstrap.cc:492] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:28.590257 17424 tablet_bootstrap.cc:654] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:28.593083 17404 tablet_bootstrap.cc:492] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:28.593533 17424 tablet_bootstrap.cc:492] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:28.593767 17424 ts_tablet_manager.cc:1367] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 08:18:28.594759 17404 tablet_bootstrap.cc:654] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:28.594972 17424 raft_consensus.cc:348] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:28.595137 17424 raft_consensus.cc:374] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:28.595196 17424 raft_consensus.cc:729] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:28.595374 17424 consensus_queue.cc:260] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:28.596088 17424 ts_tablet_manager.cc:1398] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:28.605333 17425 tablet_bootstrap.cc:492] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:28.606145 17404 tablet_bootstrap.cc:492] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:28.606319 17404 ts_tablet_manager.cc:1367] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20240502 08:18:28.607004 17425 tablet_bootstrap.cc:654] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:28.607123 17404 raft_consensus.cc:348] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:28.607443 17404 raft_consensus.cc:374] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:28.607573 17404 raft_consensus.cc:729] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:28.609462 17404 consensus_queue.cc:260] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:28.616060 17404 ts_tablet_manager.cc:1398] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 08:18:28.616416 17425 tablet_bootstrap.cc:492] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:28.616591 17425 ts_tablet_manager.cc:1367] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 08:18:28.617508 17425 raft_consensus.cc:348] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:28.617815 17425 raft_consensus.cc:374] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:28.617946 17425 raft_consensus.cc:729] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:28.618247 17425 consensus_queue.cc:260] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:28.621523 17425 ts_tablet_manager.cc:1398] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:18:28.773438 17389 consensus_queue.cc:1035] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:28.778770 17389 consensus_queue.cc:1035] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:28.860680 17398 consensus_queue.cc:1035] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:28.860792 17275 raft_consensus.cc:482] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:28.860919 17275 raft_consensus.cc:504] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:28.861595 17275 leader_election.cc:288] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:28.861900 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f582c3bcc16141e2afbf2d0cb6691f1d" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:28.862123 12727 raft_consensus.cc:2454] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:28.862090 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f582c3bcc16141e2afbf2d0cb6691f1d" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:28.862313 15486 raft_consensus.cc:2454] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:28.862859 15688 leader_election.cc:302] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:28.863340 17275 raft_consensus.cc:2781] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:28.863456 17275 raft_consensus.cc:482] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:28.863543 17275 raft_consensus.cc:3037] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:28.865830 17263 consensus_queue.cc:1035] T a5af24c28aa04f72844744771f6b7249 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:28.866101 17275 raft_consensus.cc:504] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:28.866717 17275 leader_election.cc:288] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:28.867199 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f582c3bcc16141e2afbf2d0cb6691f1d" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:28.867278 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f582c3bcc16141e2afbf2d0cb6691f1d" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:28.867412 15486 raft_consensus.cc:3037] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:28.867477 12727 raft_consensus.cc:3037] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:28.869685 12727 raft_consensus.cc:2454] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:28.870190 15688 leader_election.cc:302] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:28.870484 17275 raft_consensus.cc:2781] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:28.870625 17275 raft_consensus.cc:686] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:28.870908 17275 consensus_queue.cc:237] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:28.870421 15486 raft_consensus.cc:2454] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:28.875578 12588 catalog_manager.cc:5484] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:28.910920 17064 consensus_queue.cc:1035] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:28.915344 17064 consensus_queue.cc:1035] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:28.991415 12588 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:28.997514 15486 raft_consensus.cc:1261] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:28.997514 12727 raft_consensus.cc:1261] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:28.998118 17275 consensus_queue.cc:1035] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:28.998382 17415 consensus_queue.cc:1035] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:29.004637 12584 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet f582c3bcc16141e2afbf2d0cb6691f1d (table rebalancer_dynamic_table [id=6cacde43f7df4d8591aacc49979244a0]). Expected version 1 got 0
I20240502 08:18:29.005156 12585 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet f582c3bcc16141e2afbf2d0cb6691f1d (table rebalancer_dynamic_table [id=6cacde43f7df4d8591aacc49979244a0]). Expected version 1 got 0
I20240502 08:18:29.008183 17442 mvcc.cc:205] Tried to move back new op lower bound from 7023156875247570944 to 7023156874737250304. Current Snapshot: MvccSnapshot[applied={T|T < 7023156875247570944}]
I20240502 08:18:29.010778 17446 tablet.cc:1722] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:29.011533 17446 tablet.cc:1620] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:29.013156 17445 tablet.cc:1722] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:29.014096 17445 tablet.cc:1620] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:29.017791 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=6cacde43f7df4d8591aacc49979244a0] alter complete (version 1)
I20240502 08:18:29.019953 17447 tablet.cc:1722] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:29.020488 17447 tablet.cc:1620] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:29.023234 12585 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet f582c3bcc16141e2afbf2d0cb6691f1d (table rebalancer_dynamic_table [id=6cacde43f7df4d8591aacc49979244a0]). Expected version 1 got 0
I20240502 08:18:29.074493 12585 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:29.083508 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 109bc82d7ff94f059dcbc80555b88b1b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77da14560704c279516343b861254e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:29.083855 12710 tablet_service.cc:1467] Processing CreateTablet for tablet 109bc82d7ff94f059dcbc80555b88b1b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77da14560704c279516343b861254e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:29.084033 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109bc82d7ff94f059dcbc80555b88b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.084256 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109bc82d7ff94f059dcbc80555b88b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.087510 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 039268a812b64d88a922d0399074647c (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77da14560704c279516343b861254e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:29.087913 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039268a812b64d88a922d0399074647c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.088608 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 039268a812b64d88a922d0399074647c (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77da14560704c279516343b861254e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:29.089012 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039268a812b64d88a922d0399074647c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.091307 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 835f0b402ea24c12b036075e161233bd (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77da14560704c279516343b861254e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:29.091547 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 835f0b402ea24c12b036075e161233bd (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77da14560704c279516343b861254e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:29.091742 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835f0b402ea24c12b036075e161233bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.091940 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835f0b402ea24c12b036075e161233bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.092577 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 109bc82d7ff94f059dcbc80555b88b1b (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77da14560704c279516343b861254e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:29.092960 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109bc82d7ff94f059dcbc80555b88b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.098165 17425 tablet_bootstrap.cc:492] T 835f0b402ea24c12b036075e161233bd P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:29.099591 17449 tablet_bootstrap.cc:492] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:29.099617 17425 tablet_bootstrap.cc:654] T 835f0b402ea24c12b036075e161233bd P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:29.101218 17449 tablet_bootstrap.cc:654] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:29.103873 17453 tablet_bootstrap.cc:492] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:29.106356 17453 tablet_bootstrap.cc:654] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:29.114616 17453 tablet_bootstrap.cc:492] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:29.114832 17453 ts_tablet_manager.cc:1367] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20240502 08:18:29.116405 17453 raft_consensus.cc:348] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.116799 17453 raft_consensus.cc:374] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:29.116943 17453 raft_consensus.cc:729] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:29.117152 17453 consensus_queue.cc:260] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.117717 17453 ts_tablet_manager.cc:1398] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:29.117759 17449 tablet_bootstrap.cc:492] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:29.117969 17449 ts_tablet_manager.cc:1367] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20240502 08:18:29.118819 17449 raft_consensus.cc:348] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.119113 17449 raft_consensus.cc:374] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:29.119248 17449 raft_consensus.cc:729] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:29.119547 17449 consensus_queue.cc:260] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.121030 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 835f0b402ea24c12b036075e161233bd (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77da14560704c279516343b861254e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:29.121484 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835f0b402ea24c12b036075e161233bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.122903 17449 ts_tablet_manager.cc:1398] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:29.123565 12585 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:29.125257 17425 tablet_bootstrap.cc:492] T 835f0b402ea24c12b036075e161233bd P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:29.125416 17425 ts_tablet_manager.cc:1367] T 835f0b402ea24c12b036075e161233bd P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240502 08:18:29.126116 17425 raft_consensus.cc:348] T 835f0b402ea24c12b036075e161233bd P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.126322 17425 raft_consensus.cc:374] T 835f0b402ea24c12b036075e161233bd P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:29.127115 17425 raft_consensus.cc:729] T 835f0b402ea24c12b036075e161233bd P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:29.127409 17425 consensus_queue.cc:260] T 835f0b402ea24c12b036075e161233bd P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.128762 17425 ts_tablet_manager.cc:1398] T 835f0b402ea24c12b036075e161233bd P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:29.133867 17445 tablet.cc:1722] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:29.134353 17445 tablet.cc:1620] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:29.136772 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 039268a812b64d88a922d0399074647c (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b77da14560704c279516343b861254e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:29.137262 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039268a812b64d88a922d0399074647c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.143898 17447 tablet.cc:1722] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:29.144369 17447 tablet.cc:1620] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:29.148533 17446 tablet.cc:1722] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:29.149040 17446 tablet.cc:1620] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:29.150161 12585 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=6cacde43f7df4d8591aacc49979244a0] alter complete (version 2)
I20240502 08:18:29.155026 17459 tablet_bootstrap.cc:492] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:29.158195 17458 tablet_bootstrap.cc:492] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:29.160133 17369 raft_consensus.cc:482] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:29.160310 17369 raft_consensus.cc:504] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.161203 17369 leader_election.cc:288] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:29.161643 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109bc82d7ff94f059dcbc80555b88b1b" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:29.161850 15886 raft_consensus.cc:2454] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:29.162207 17459 tablet_bootstrap.cc:654] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:29.162936 17458 tablet_bootstrap.cc:654] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:29.164115 13057 leader_election.cc:302] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
W20240502 08:18:29.164412 13056 leader_election.cc:341] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Not found: Tablet not found: 109bc82d7ff94f059dcbc80555b88b1b
I20240502 08:18:29.166072 17369 raft_consensus.cc:2781] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:29.166174 17369 raft_consensus.cc:482] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:29.166266 17369 raft_consensus.cc:3037] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:29.168843 17369 raft_consensus.cc:504] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.169932 17369 leader_election.cc:288] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:29.169934 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109bc82d7ff94f059dcbc80555b88b1b" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:29.170145 15887 raft_consensus.cc:3037] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:29.171610 17460 tablet_bootstrap.cc:492] T 835f0b402ea24c12b036075e161233bd P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:29.171948 17451 tablet_bootstrap.cc:492] T 039268a812b64d88a922d0399074647c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:29.172749 15887 raft_consensus.cc:2454] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:29.173153 17451 tablet_bootstrap.cc:654] T 039268a812b64d88a922d0399074647c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:29.173174 17460 tablet_bootstrap.cc:654] T 835f0b402ea24c12b036075e161233bd P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:29.174362 13057 leader_election.cc:302] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:29.174438 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109bc82d7ff94f059dcbc80555b88b1b" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
W20240502 08:18:29.174921 13056 leader_election.cc:341] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:29.175073 17459 tablet_bootstrap.cc:492] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:29.175210 17459 ts_tablet_manager.cc:1367] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 08:18:29.175251 17369 raft_consensus.cc:2781] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:29.175386 17369 raft_consensus.cc:686] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:18:29.175617 17369 consensus_queue.cc:237] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.175922 17459 raft_consensus.cc:348] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.176107 17459 raft_consensus.cc:374] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:29.176178 17459 raft_consensus.cc:729] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:29.176419 17459 consensus_queue.cc:260] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.177054 17459 ts_tablet_manager.cc:1398] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:29.184113 12588 catalog_manager.cc:5484] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:29.184736 17458 tablet_bootstrap.cc:492] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:29.184870 17458 ts_tablet_manager.cc:1367] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20240502 08:18:29.186125 17460 tablet_bootstrap.cc:492] T 835f0b402ea24c12b036075e161233bd P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:29.186443 17460 ts_tablet_manager.cc:1367] T 835f0b402ea24c12b036075e161233bd P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20240502 08:18:29.186363 17458 raft_consensus.cc:348] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.186606 17458 raft_consensus.cc:374] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:29.186695 17458 raft_consensus.cc:729] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:29.186933 17458 consensus_queue.cc:260] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.187497 17458 ts_tablet_manager.cc:1398] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:29.190160 17460 raft_consensus.cc:348] T 835f0b402ea24c12b036075e161233bd P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.190555 17460 raft_consensus.cc:374] T 835f0b402ea24c12b036075e161233bd P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:29.190742 17460 raft_consensus.cc:729] T 835f0b402ea24c12b036075e161233bd P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:29.191121 17460 consensus_queue.cc:260] T 835f0b402ea24c12b036075e161233bd P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.193089 17460 ts_tablet_manager.cc:1398] T 835f0b402ea24c12b036075e161233bd P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:18:29.193429 17460 tablet_bootstrap.cc:492] T 039268a812b64d88a922d0399074647c P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:29.193707 17451 tablet_bootstrap.cc:492] T 039268a812b64d88a922d0399074647c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:29.195217 17451 ts_tablet_manager.cc:1367] T 039268a812b64d88a922d0399074647c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.005s
I20240502 08:18:29.196189 17460 tablet_bootstrap.cc:654] T 039268a812b64d88a922d0399074647c P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:29.196030 17451 raft_consensus.cc:348] T 039268a812b64d88a922d0399074647c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.196359 17451 raft_consensus.cc:374] T 039268a812b64d88a922d0399074647c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:29.197109 17451 raft_consensus.cc:729] T 039268a812b64d88a922d0399074647c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:29.197420 17451 consensus_queue.cc:260] T 039268a812b64d88a922d0399074647c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.199079 17451 ts_tablet_manager.cc:1398] T 039268a812b64d88a922d0399074647c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:29.204156 17460 tablet_bootstrap.cc:492] T 039268a812b64d88a922d0399074647c P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:29.204389 17460 ts_tablet_manager.cc:1367] T 039268a812b64d88a922d0399074647c P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 08:18:29.205386 17460 raft_consensus.cc:348] T 039268a812b64d88a922d0399074647c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.205917 17460 raft_consensus.cc:374] T 039268a812b64d88a922d0399074647c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:29.206012 17460 raft_consensus.cc:729] T 039268a812b64d88a922d0399074647c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:29.206293 17460 consensus_queue.cc:260] T 039268a812b64d88a922d0399074647c P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.206995 17460 ts_tablet_manager.cc:1398] T 039268a812b64d88a922d0399074647c P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:29.207515 17460 tablet_bootstrap.cc:492] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:29.209240 17460 tablet_bootstrap.cc:654] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:29.217694 17460 tablet_bootstrap.cc:492] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:29.217906 17460 ts_tablet_manager.cc:1367] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20240502 08:18:29.218627 17460 raft_consensus.cc:348] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.218788 17460 raft_consensus.cc:374] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:29.218879 17460 raft_consensus.cc:729] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:29.219146 17460 consensus_queue.cc:260] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.219624 17460 ts_tablet_manager.cc:1398] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:29.246587 17275 raft_consensus.cc:482] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:29.246790 17275 raft_consensus.cc:504] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.247659 17275 leader_election.cc:288] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:29.248061 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835f0b402ea24c12b036075e161233bd" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:29.248198 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835f0b402ea24c12b036075e161233bd" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:29.248289 12730 raft_consensus.cc:2454] T 835f0b402ea24c12b036075e161233bd P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:29.248451 15486 raft_consensus.cc:2454] T 835f0b402ea24c12b036075e161233bd P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:29.249177 15688 leader_election.cc:302] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:29.249450 17275 raft_consensus.cc:2781] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:29.249574 17275 raft_consensus.cc:482] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:29.249679 17275 raft_consensus.cc:3037] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:29.251971 17275 raft_consensus.cc:504] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.252497 17275 leader_election.cc:288] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:29.252900 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835f0b402ea24c12b036075e161233bd" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:29.252991 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835f0b402ea24c12b036075e161233bd" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:29.253082 12730 raft_consensus.cc:3037] T 835f0b402ea24c12b036075e161233bd P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:29.253183 15486 raft_consensus.cc:3037] T 835f0b402ea24c12b036075e161233bd P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:29.255453 12730 raft_consensus.cc:2454] T 835f0b402ea24c12b036075e161233bd P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:29.255717 15486 raft_consensus.cc:2454] T 835f0b402ea24c12b036075e161233bd P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:29.256080 15688 leader_election.cc:302] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:29.256340 17275 raft_consensus.cc:2781] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:29.256477 17275 raft_consensus.cc:686] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:29.256763 17275 consensus_queue.cc:237] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.257674 12585 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:29.261015 12588 catalog_manager.cc:5484] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:29.270875 17445 tablet.cc:1722] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:29.282864 17446 tablet.cc:1722] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:29.283553 17447 tablet.cc:1722] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:29.290863 12588 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=6cacde43f7df4d8591aacc49979244a0] alter complete (version 3)
I20240502 08:18:29.393460 12585 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:29.393715 12585 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:29.396698 12585 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet f582c3bcc16141e2afbf2d0cb6691f1d
I20240502 08:18:29.397501 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet f582c3bcc16141e2afbf2d0cb6691f1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:29 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:29.397507 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet f582c3bcc16141e2afbf2d0cb6691f1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:29 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:29.397665 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet f582c3bcc16141e2afbf2d0cb6691f1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:29 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:29.398162 17471 tablet_replica.cc:330] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:29.398353 17471 raft_consensus.cc:2229] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:29.398614 17471 raft_consensus.cc:2258] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:29.399544 17471 ts_tablet_manager.cc:1875] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:29.404778 17471 ts_tablet_manager.cc:1888] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:29.404978 17471 log.cc:1196] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/f582c3bcc16141e2afbf2d0cb6691f1d
I20240502 08:18:29.405422 17471 ts_tablet_manager.cc:1909] T f582c3bcc16141e2afbf2d0cb6691f1d P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:29.406663 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet f582c3bcc16141e2afbf2d0cb6691f1d (table rebalancer_dynamic_table_new_name [id=6cacde43f7df4d8591aacc49979244a0]) successfully deleted
I20240502 08:18:29.409873 17472 tablet_replica.cc:330] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:29.410562 17473 tablet_replica.cc:330] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:29.412055 17473 raft_consensus.cc:2229] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:29.412097 17472 raft_consensus.cc:2229] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:29.412340 17473 raft_consensus.cc:2258] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:29.412508 17472 raft_consensus.cc:2258] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:29.413190 17473 ts_tablet_manager.cc:1875] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:29.413398 17472 ts_tablet_manager.cc:1875] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:29.419890 17472 ts_tablet_manager.cc:1888] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:29.419890 17473 ts_tablet_manager.cc:1888] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:29.420025 17473 log.cc:1196] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/f582c3bcc16141e2afbf2d0cb6691f1d
I20240502 08:18:29.420025 17472 log.cc:1196] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/f582c3bcc16141e2afbf2d0cb6691f1d
I20240502 08:18:29.420500 17473 ts_tablet_manager.cc:1909] T f582c3bcc16141e2afbf2d0cb6691f1d P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:29.420495 17472 ts_tablet_manager.cc:1909] T f582c3bcc16141e2afbf2d0cb6691f1d P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:29.421554 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet f582c3bcc16141e2afbf2d0cb6691f1d (table rebalancer_dynamic_table_new_name [id=6cacde43f7df4d8591aacc49979244a0]) successfully deleted
I20240502 08:18:29.421662 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet f582c3bcc16141e2afbf2d0cb6691f1d (table rebalancer_dynamic_table_new_name [id=6cacde43f7df4d8591aacc49979244a0]) successfully deleted
I20240502 08:18:29.485229 17389 raft_consensus.cc:482] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:29.485438 17389 raft_consensus.cc:504] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.486173 17389 leader_election.cc:288] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:29.486727 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039268a812b64d88a922d0399074647c" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:29.486822 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039268a812b64d88a922d0399074647c" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:29.486992 12730 raft_consensus.cc:2454] T 039268a812b64d88a922d0399074647c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:29.487047 15620 raft_consensus.cc:2454] T 039268a812b64d88a922d0399074647c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:29.487490 12798 leader_election.cc:302] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:29.487785 17389 raft_consensus.cc:2781] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:29.487871 17389 raft_consensus.cc:482] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:29.487974 17389 raft_consensus.cc:3037] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:29.490212 17389 raft_consensus.cc:504] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.490921 17389 leader_election.cc:288] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:29.491202 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039268a812b64d88a922d0399074647c" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:29.491391 15620 raft_consensus.cc:3037] T 039268a812b64d88a922d0399074647c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:29.491402 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039268a812b64d88a922d0399074647c" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:29.491591 12730 raft_consensus.cc:3037] T 039268a812b64d88a922d0399074647c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:29.494174 15620 raft_consensus.cc:2454] T 039268a812b64d88a922d0399074647c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:29.494166 12730 raft_consensus.cc:2454] T 039268a812b64d88a922d0399074647c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:29.494603 12798 leader_election.cc:302] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:29.494905 17389 raft_consensus.cc:2781] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:29.495383 17389 raft_consensus.cc:686] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:18:29.495648 17389 consensus_queue.cc:237] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.499389 12588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:29.499410 12585 catalog_manager.cc:5484] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:29.506573 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 119beb2a2c2b4afc917727c89d6fb396 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=22e48306960b4f039d1ee9f0c2a48ea3]), partition=
I20240502 08:18:29.506955 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119beb2a2c2b4afc917727c89d6fb396. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.507012 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 119beb2a2c2b4afc917727c89d6fb396 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=22e48306960b4f039d1ee9f0c2a48ea3]), partition=
I20240502 08:18:29.507386 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119beb2a2c2b4afc917727c89d6fb396. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.511071 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 119beb2a2c2b4afc917727c89d6fb396 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=22e48306960b4f039d1ee9f0c2a48ea3]), partition=
I20240502 08:18:29.511428 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 119beb2a2c2b4afc917727c89d6fb396. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.516121 17476 tablet_bootstrap.cc:492] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:29.517529 17476 tablet_bootstrap.cc:654] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:29.520233 17476 tablet_bootstrap.cc:492] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:29.520363 17476 ts_tablet_manager.cc:1367] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:18:29.520646 17425 tablet_bootstrap.cc:492] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:29.520712 17458 tablet_bootstrap.cc:492] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:29.521097 17476 raft_consensus.cc:348] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.521320 17476 raft_consensus.cc:374] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:29.521397 17476 raft_consensus.cc:729] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:29.521605 17476 consensus_queue.cc:260] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.521858 17458 tablet_bootstrap.cc:654] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:29.522055 17425 tablet_bootstrap.cc:654] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:29.522114 17476 ts_tablet_manager.cc:1398] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:29.524503 17458 tablet_bootstrap.cc:492] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:29.524646 17458 ts_tablet_manager.cc:1367] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:29.525245 17458 raft_consensus.cc:348] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.525429 17458 raft_consensus.cc:374] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:29.525492 17458 raft_consensus.cc:729] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:29.525708 17458 consensus_queue.cc:260] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.526238 17458 ts_tablet_manager.cc:1398] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:29.535418 17425 tablet_bootstrap.cc:492] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:29.535538 17425 ts_tablet_manager.cc:1367] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20240502 08:18:29.536075 17425 raft_consensus.cc:348] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.536309 17425 raft_consensus.cc:374] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:29.536381 17425 raft_consensus.cc:729] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:29.536549 17425 consensus_queue.cc:260] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.537326 17425 ts_tablet_manager.cc:1398] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:29.589908 17064 raft_consensus.cc:482] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:29.590119 17064 raft_consensus.cc:504] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.591189 17064 leader_election.cc:288] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:29.591768 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119beb2a2c2b4afc917727c89d6fb396" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:29.591833 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119beb2a2c2b4afc917727c89d6fb396" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:29.592000 15486 raft_consensus.cc:2454] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:29.592038 15755 raft_consensus.cc:2454] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:29.592514 12926 leader_election.cc:302] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:29.593134 17064 raft_consensus.cc:2781] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:29.593228 17064 raft_consensus.cc:482] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:29.593328 17064 raft_consensus.cc:3037] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:29.595278 17064 raft_consensus.cc:504] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.595782 17064 leader_election.cc:288] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:29.596138 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119beb2a2c2b4afc917727c89d6fb396" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:29.596362 15755 raft_consensus.cc:3037] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:29.596325 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "119beb2a2c2b4afc917727c89d6fb396" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:29.596499 15486 raft_consensus.cc:3037] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:29.598243 15755 raft_consensus.cc:2454] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:29.598642 12925 leader_election.cc:302] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:29.598917 17064 raft_consensus.cc:2781] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:29.599095 17064 raft_consensus.cc:686] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:29.599355 17064 consensus_queue.cc:237] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:29.602842 15486 raft_consensus.cc:2454] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:29.606521 17154 raft_consensus.cc:482] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:29.606652 17154 raft_consensus.cc:504] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:29.607343 17154 leader_election.cc:288] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:29.607643 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109bc82d7ff94f059dcbc80555b88b1b" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:29.608120 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109bc82d7ff94f059dcbc80555b88b1b" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:29.608358 15886 raft_consensus.cc:2379] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29b035434087435cbaf4ebcac98eedee in current term 1: Already voted for candidate 6a0b05f3250d415590b0edf6b44fbe1f in this term.
I20240502 08:18:29.608701 12665 leader_election.cc:302] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee; no voters: 10f0e36203914b85b77e78609ffced98, 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:18:29.608986 17154 raft_consensus.cc:3037] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:29.610992 17154 raft_consensus.cc:2726] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:29.623836 12585 catalog_manager.cc:5484] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:29.676662 17347 consensus_queue.cc:1035] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:29.684278 17483 consensus_queue.cc:1035] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:29.690891 17470 consensus_queue.cc:1035] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:29.695474 17275 consensus_queue.cc:1035] T 835f0b402ea24c12b036075e161233bd P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:29.736939 12585 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:29.743248 15754 raft_consensus.cc:1261] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:29.743266 15486 raft_consensus.cc:1261] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:29.743767 17064 consensus_queue.cc:1035] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:29.744045 17346 consensus_queue.cc:1035] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:29.748656 17491 tablet.cc:1722] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:29.748761 12588 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet 119beb2a2c2b4afc917727c89d6fb396 (table rebalancer_dynamic_table [id=22e48306960b4f039d1ee9f0c2a48ea3]). Expected version 1 got 0
I20240502 08:18:29.749574 17432 mvcc.cc:205] Tried to move back new op lower bound from 7023156878302875648 to 7023156877723312128. Current Snapshot: MvccSnapshot[applied={T|T < 7023156878302875648}]
I20240502 08:18:29.749781 17491 tablet.cc:1620] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:29.749780 17445 tablet.cc:1722] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:29.750180 17445 tablet.cc:1620] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:29.760614 12584 catalog_manager.cc:5576] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131) does not have the latest schema for tablet 119beb2a2c2b4afc917727c89d6fb396 (table rebalancer_dynamic_table [id=22e48306960b4f039d1ee9f0c2a48ea3]). Expected version 1 got 0
I20240502 08:18:29.762845 12588 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=22e48306960b4f039d1ee9f0c2a48ea3] alter complete (version 1)
I20240502 08:18:29.764501 17446 tablet.cc:1722] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:29.764973 17446 tablet.cc:1620] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:29.869639 12585 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:29.877453 17446 tablet.cc:1722] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:29.877915 17446 tablet.cc:1620] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:29.877969 17445 tablet.cc:1722] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:29.878448 17445 tablet.cc:1620] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:29.879501 17389 consensus_queue.cc:1035] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:29.876994 17491 tablet.cc:1722] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:29.887717 17491 tablet.cc:1620] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:29.888456 12585 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=22e48306960b4f039d1ee9f0c2a48ea3] alter complete (version 2)
I20240502 08:18:29.896203 17389 consensus_queue.cc:1035] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:29.996568 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:30.003366 17491 tablet.cc:1722] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:30.003963 17446 tablet.cc:1722] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:30.004931 17445 tablet.cc:1722] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:30.014385 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=22e48306960b4f039d1ee9f0c2a48ea3] alter complete (version 3)
I20240502 08:18:30.066752 12585 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:30.075443 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 28f7285fe2dc4c5ab0520c39ae994793 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b251c3886b9548c2bbc30da21fc3f84b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:30.075784 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 28f7285fe2dc4c5ab0520c39ae994793 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b251c3886b9548c2bbc30da21fc3f84b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:30.075928 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f7285fe2dc4c5ab0520c39ae994793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.076160 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f7285fe2dc4c5ab0520c39ae994793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.076285 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 24c29ddfac024726ad9ee74d77c115f9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b251c3886b9548c2bbc30da21fc3f84b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:30.076685 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c29ddfac024726ad9ee74d77c115f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.081422 17497 tablet_bootstrap.cc:492] T 24c29ddfac024726ad9ee74d77c115f9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:30.081630 15467 tablet_service.cc:1467] Processing CreateTablet for tablet 24c29ddfac024726ad9ee74d77c115f9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b251c3886b9548c2bbc30da21fc3f84b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:30.082003 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c29ddfac024726ad9ee74d77c115f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.082044 15866 tablet_service.cc:1467] Processing CreateTablet for tablet b73852b46f7e4db89e30d83edafa6413 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b251c3886b9548c2bbc30da21fc3f84b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:30.082445 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73852b46f7e4db89e30d83edafa6413. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.083130 17497 tablet_bootstrap.cc:654] T 24c29ddfac024726ad9ee74d77c115f9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:30.083416 15733 tablet_service.cc:1467] Processing CreateTablet for tablet b73852b46f7e4db89e30d83edafa6413 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b251c3886b9548c2bbc30da21fc3f84b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:30.083792 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73852b46f7e4db89e30d83edafa6413. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.086866 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 24c29ddfac024726ad9ee74d77c115f9 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b251c3886b9548c2bbc30da21fc3f84b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:30.087258 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24c29ddfac024726ad9ee74d77c115f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.089890 17501 tablet_bootstrap.cc:492] T 28f7285fe2dc4c5ab0520c39ae994793 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:30.090874 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 28f7285fe2dc4c5ab0520c39ae994793 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b251c3886b9548c2bbc30da21fc3f84b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:30.091300 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28f7285fe2dc4c5ab0520c39ae994793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.094409 17498 tablet_bootstrap.cc:492] T 28f7285fe2dc4c5ab0520c39ae994793 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:30.098207 17503 tablet_bootstrap.cc:492] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:30.098351 17498 tablet_bootstrap.cc:654] T 28f7285fe2dc4c5ab0520c39ae994793 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:30.099452 17502 tablet_bootstrap.cc:492] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:30.099601 17501 tablet_bootstrap.cc:654] T 28f7285fe2dc4c5ab0520c39ae994793 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:30.101629 12970 tablet_service.cc:1467] Processing CreateTablet for tablet b73852b46f7e4db89e30d83edafa6413 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=b251c3886b9548c2bbc30da21fc3f84b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:30.102046 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73852b46f7e4db89e30d83edafa6413. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.102517 17500 tablet_bootstrap.cc:492] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:30.106585 17506 tablet_bootstrap.cc:492] T 24c29ddfac024726ad9ee74d77c115f9 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:30.106923 17500 tablet_bootstrap.cc:654] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:30.107928 17503 tablet_bootstrap.cc:654] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:30.108239 17506 tablet_bootstrap.cc:654] T 24c29ddfac024726ad9ee74d77c115f9 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:30.108505 17502 tablet_bootstrap.cc:654] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:30.109614 17498 tablet_bootstrap.cc:492] T 28f7285fe2dc4c5ab0520c39ae994793 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:30.109805 17498 ts_tablet_manager.cc:1367] T 28f7285fe2dc4c5ab0520c39ae994793 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 08:18:30.109897 17501 tablet_bootstrap.cc:492] T 28f7285fe2dc4c5ab0520c39ae994793 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:30.110622 17497 tablet_bootstrap.cc:492] T 24c29ddfac024726ad9ee74d77c115f9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:30.110551 17498 raft_consensus.cc:348] T 28f7285fe2dc4c5ab0520c39ae994793 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.110806 17497 ts_tablet_manager.cc:1367] T 24c29ddfac024726ad9ee74d77c115f9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.003s
I20240502 08:18:30.110841 17498 raft_consensus.cc:374] T 28f7285fe2dc4c5ab0520c39ae994793 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:30.110999 17498 raft_consensus.cc:729] T 28f7285fe2dc4c5ab0520c39ae994793 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:30.111083 17503 tablet_bootstrap.cc:492] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:30.111253 17503 ts_tablet_manager.cc:1367] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 08:18:30.111274 17498 consensus_queue.cc:260] T 28f7285fe2dc4c5ab0520c39ae994793 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.111490 17500 tablet_bootstrap.cc:492] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:30.111797 17500 ts_tablet_manager.cc:1367] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240502 08:18:30.111873 17498 ts_tablet_manager.cc:1398] T 28f7285fe2dc4c5ab0520c39ae994793 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:30.111970 17503 raft_consensus.cc:348] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.111629 17497 raft_consensus.cc:348] T 24c29ddfac024726ad9ee74d77c115f9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:30.112237 17503 raft_consensus.cc:374] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:30.112293 17497 raft_consensus.cc:374] T 24c29ddfac024726ad9ee74d77c115f9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:30.112363 17503 raft_consensus.cc:729] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:30.112444 17497 raft_consensus.cc:729] T 24c29ddfac024726ad9ee74d77c115f9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:30.113853 17503 consensus_queue.cc:260] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.114450 17503 ts_tablet_manager.cc:1398] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:30.114787 17503 tablet_bootstrap.cc:492] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:30.112712 17497 consensus_queue.cc:260] T 24c29ddfac024726ad9ee74d77c115f9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:30.116925 17497 ts_tablet_manager.cc:1398] T 24c29ddfac024726ad9ee74d77c115f9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:30.117203 17501 ts_tablet_manager.cc:1367] T 28f7285fe2dc4c5ab0520c39ae994793 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240502 08:18:30.117898 17503 tablet_bootstrap.cc:654] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:30.118052 17501 raft_consensus.cc:348] T 28f7285fe2dc4c5ab0520c39ae994793 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.118322 17501 raft_consensus.cc:374] T 28f7285fe2dc4c5ab0520c39ae994793 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:30.118453 17501 raft_consensus.cc:729] T 28f7285fe2dc4c5ab0520c39ae994793 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:30.118777 17501 consensus_queue.cc:260] T 28f7285fe2dc4c5ab0520c39ae994793 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.119431 17501 ts_tablet_manager.cc:1398] T 28f7285fe2dc4c5ab0520c39ae994793 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:30.123600 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:30.123816 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:30.128512 17503 tablet_bootstrap.cc:492] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:30.128718 17503 ts_tablet_manager.cc:1367] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.008s
I20240502 08:18:30.129480 17503 raft_consensus.cc:348] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.129752 17503 raft_consensus.cc:374] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:30.129884 17503 raft_consensus.cc:729] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:30.130515 17503 consensus_queue.cc:260] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.130838 17506 tablet_bootstrap.cc:492] T 24c29ddfac024726ad9ee74d77c115f9 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:30.131021 17506 ts_tablet_manager.cc:1367] T 24c29ddfac024726ad9ee74d77c115f9 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.006s
I20240502 08:18:30.131819 17506 raft_consensus.cc:348] T 24c29ddfac024726ad9ee74d77c115f9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:30.132110 17506 raft_consensus.cc:374] T 24c29ddfac024726ad9ee74d77c115f9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:30.132251 17506 raft_consensus.cc:729] T 24c29ddfac024726ad9ee74d77c115f9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:30.132534 17506 consensus_queue.cc:260] T 24c29ddfac024726ad9ee74d77c115f9 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:30.135124 17503 ts_tablet_manager.cc:1398] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 08:18:30.138518 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 119beb2a2c2b4afc917727c89d6fb396
I20240502 08:18:30.139230 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 119beb2a2c2b4afc917727c89d6fb396 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:30 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:30.140148 17514 tablet_replica.cc:330] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:30.140362 17514 raft_consensus.cc:2229] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:30.140671 17514 raft_consensus.cc:2258] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:30.141036 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 119beb2a2c2b4afc917727c89d6fb396 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:30 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:30.141182 15467 tablet_service.cc:1514] Processing DeleteTablet for tablet 119beb2a2c2b4afc917727c89d6fb396 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:30 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:30.142906 17506 ts_tablet_manager.cc:1398] T 24c29ddfac024726ad9ee74d77c115f9 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.012s user 0.000s sys 0.001s
I20240502 08:18:30.148694 17516 tablet_replica.cc:330] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:30.148722 17514 ts_tablet_manager.cc:1875] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:30.148885 17516 raft_consensus.cc:2229] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:30.149078 17516 raft_consensus.cc:2258] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:30.149708 17516 ts_tablet_manager.cc:1875] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:30.112674 17500 raft_consensus.cc:348] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.154500 17500 raft_consensus.cc:374] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:30.154662 17500 raft_consensus.cc:729] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:30.154964 17500 consensus_queue.cc:260] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.155735 17500 ts_tablet_manager.cc:1398] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.044s user 0.002s sys 0.000s
I20240502 08:18:30.155323 17516 ts_tablet_manager.cc:1888] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:30.157809 17502 tablet_bootstrap.cc:492] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:30.157961 17502 ts_tablet_manager.cc:1367] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.000s
I20240502 08:18:30.158772 17516 log.cc:1196] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/119beb2a2c2b4afc917727c89d6fb396
I20240502 08:18:30.159137 17516 ts_tablet_manager.cc:1909] T 119beb2a2c2b4afc917727c89d6fb396 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:30.159328 17515 tablet_replica.cc:330] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:30.159533 17515 raft_consensus.cc:2229] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:30.159691 17500 tablet_bootstrap.cc:492] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:30.159934 17515 raft_consensus.cc:2258] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:30.160667 17515 ts_tablet_manager.cc:1875] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:30.160667 17502 raft_consensus.cc:348] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.160933 17502 raft_consensus.cc:374] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:30.161074 17502 raft_consensus.cc:729] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:30.161072 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 119beb2a2c2b4afc917727c89d6fb396 (table rebalancer_dynamic_table_new_name [id=22e48306960b4f039d1ee9f0c2a48ea3]) successfully deleted
I20240502 08:18:30.161404 17502 consensus_queue.cc:260] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.161821 17500 tablet_bootstrap.cc:654] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:30.163547 17502 ts_tablet_manager.cc:1398] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:30.162178 17514 ts_tablet_manager.cc:1888] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:30.167732 17514 log.cc:1196] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/119beb2a2c2b4afc917727c89d6fb396
I20240502 08:18:30.168220 17514 ts_tablet_manager.cc:1909] T 119beb2a2c2b4afc917727c89d6fb396 P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:30.171519 17515 ts_tablet_manager.cc:1888] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:30.171625 17515 log.cc:1196] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/119beb2a2c2b4afc917727c89d6fb396
I20240502 08:18:30.172091 17515 ts_tablet_manager.cc:1909] T 119beb2a2c2b4afc917727c89d6fb396 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:30.172161 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 119beb2a2c2b4afc917727c89d6fb396 (table rebalancer_dynamic_table_new_name [id=22e48306960b4f039d1ee9f0c2a48ea3]) successfully deleted
I20240502 08:18:30.172957 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 119beb2a2c2b4afc917727c89d6fb396 (table rebalancer_dynamic_table_new_name [id=22e48306960b4f039d1ee9f0c2a48ea3]) successfully deleted
I20240502 08:18:30.174109 17500 tablet_bootstrap.cc:492] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:30.174337 17500 ts_tablet_manager.cc:1367] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20240502 08:18:30.175187 17500 raft_consensus.cc:348] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:30.175416 17500 raft_consensus.cc:374] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:30.175503 17500 raft_consensus.cc:729] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:30.176141 17500 consensus_queue.cc:260] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:30.176622 17500 ts_tablet_manager.cc:1398] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:30.242007 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:30.248596 17275 raft_consensus.cc:482] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:30.248775 17275 raft_consensus.cc:504] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.249500 17275 leader_election.cc:288] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:30.250526 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73852b46f7e4db89e30d83edafa6413" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:30.250747 15887 raft_consensus.cc:2454] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:30.250844 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73852b46f7e4db89e30d83edafa6413" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:30.251025 12990 raft_consensus.cc:2454] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:30.251466 15688 leader_election.cc:302] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:30.252061 17415 raft_consensus.cc:2781] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:30.252189 17415 raft_consensus.cc:482] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:30.252276 17415 raft_consensus.cc:3037] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:30.252270 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78cc96594e3b4757b71f3cd47a54fd51]), partition=
I20240502 08:18:30.252367 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78cc96594e3b4757b71f3cd47a54fd51]), partition=
I20240502 08:18:30.252626 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.252723 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.255303 17415 raft_consensus.cc:504] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.256778 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73852b46f7e4db89e30d83edafa6413" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:30.256965 12990 raft_consensus.cc:3037] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:30.257592 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b73852b46f7e4db89e30d83edafa6413" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:30.257769 15887 raft_consensus.cc:3037] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:30.258531 17498 tablet_bootstrap.cc:492] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:30.258972 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78cc96594e3b4757b71f3cd47a54fd51]), partition=
I20240502 08:18:30.259312 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.259707 17498 tablet_bootstrap.cc:654] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:30.260356 15887 raft_consensus.cc:2454] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:30.261441 15689 leader_election.cc:302] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:30.262137 17520 raft_consensus.cc:2781] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:30.262365 17520 raft_consensus.cc:686] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:30.262501 17498 tablet_bootstrap.cc:492] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:30.262631 17498 ts_tablet_manager.cc:1367] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:18:30.262953 17520 consensus_queue.cc:237] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.263393 17498 raft_consensus.cc:348] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:30.263578 17498 raft_consensus.cc:374] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:30.263641 17498 raft_consensus.cc:729] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:30.263845 17498 consensus_queue.cc:260] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:30.264303 17498 ts_tablet_manager.cc:1398] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:30.264633 17457 raft_consensus.cc:482] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:30.264775 17457 raft_consensus.cc:504] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:30.265938 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c29ddfac024726ad9ee74d77c115f9" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:30.266150 17457 leader_election.cc:288] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:30.266166 15621 raft_consensus.cc:2454] T 24c29ddfac024726ad9ee74d77c115f9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:30.266633 15822 leader_election.cc:302] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:30.267197 17423 raft_consensus.cc:2781] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:30.267278 17423 raft_consensus.cc:482] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:30.267392 17423 raft_consensus.cc:3037] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:30.269934 17423 raft_consensus.cc:504] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:30.271265 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c29ddfac024726ad9ee74d77c115f9" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:30.271498 15621 raft_consensus.cc:3037] T 24c29ddfac024726ad9ee74d77c115f9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:30.272238 17423 leader_election.cc:288] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:30.274092 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c29ddfac024726ad9ee74d77c115f9" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:30.272558 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24c29ddfac024726ad9ee74d77c115f9" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:30.274327 15486 raft_consensus.cc:2454] T 24c29ddfac024726ad9ee74d77c115f9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:30.277673 12990 raft_consensus.cc:2454] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:30.285120 17415 leader_election.cc:288] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:30.274024 15621 raft_consensus.cc:2454] T 24c29ddfac024726ad9ee74d77c115f9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:30.285749 15822 leader_election.cc:302] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:30.287231 17501 tablet_bootstrap.cc:492] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:30.288692 17501 tablet_bootstrap.cc:654] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:30.288731 17522 tablet_bootstrap.cc:492] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:30.289744 17423 raft_consensus.cc:2781] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:30.290177 17423 raft_consensus.cc:686] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:30.290450 17423 consensus_queue.cc:237] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:30.291496 17522 tablet_bootstrap.cc:654] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:30.302491 17501 tablet_bootstrap.cc:492] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:30.302690 17501 ts_tablet_manager.cc:1367] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20240502 08:18:30.304777 17522 tablet_bootstrap.cc:492] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:30.304904 17522 ts_tablet_manager.cc:1367] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.005s
I20240502 08:18:30.305512 17522 raft_consensus.cc:348] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:30.306169 17501 raft_consensus.cc:348] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:30.306344 17501 raft_consensus.cc:374] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:30.306885 17522 raft_consensus.cc:374] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:30.306980 17522 raft_consensus.cc:729] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:30.307446 17522 consensus_queue.cc:260] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:30.306458 17501 raft_consensus.cc:729] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:30.308240 17501 consensus_queue.cc:260] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:30.308434 17522 ts_tablet_manager.cc:1398] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:30.310088 17501 ts_tablet_manager.cc:1398] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240502 08:18:30.323807 12588 catalog_manager.cc:5484] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:30.333290 12588 catalog_manager.cc:5484] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:30.381738 17369 raft_consensus.cc:482] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:30.381887 17369 raft_consensus.cc:504] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:30.382570 17369 leader_election.cc:288] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:30.382896 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c15a5b0a3ac4ae7bebc65b7a371dfac" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:30.382915 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c15a5b0a3ac4ae7bebc65b7a371dfac" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:30.383076 12730 raft_consensus.cc:2454] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:30.383103 12861 raft_consensus.cc:2454] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:30.383496 13056 leader_election.cc:302] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:30.383759 17369 raft_consensus.cc:2781] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:30.383854 17369 raft_consensus.cc:482] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:30.383940 17369 raft_consensus.cc:3037] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:30.386029 17369 raft_consensus.cc:504] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:30.386530 17369 leader_election.cc:288] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:30.386860 12730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c15a5b0a3ac4ae7bebc65b7a371dfac" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:30.386929 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c15a5b0a3ac4ae7bebc65b7a371dfac" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:30.387038 12730 raft_consensus.cc:3037] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:30.387094 12861 raft_consensus.cc:3037] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:30.389087 12730 raft_consensus.cc:2454] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:30.389097 12861 raft_consensus.cc:2454] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:30.389658 13056 leader_election.cc:302] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:30.389964 17369 raft_consensus.cc:2781] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:30.390115 17369 raft_consensus.cc:686] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:18:30.390398 17369 consensus_queue.cc:237] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:30.395190 12588 catalog_manager.cc:5484] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:30.420894 17064 raft_consensus.cc:482] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:30.421073 17064 raft_consensus.cc:504] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.421715 17064 leader_election.cc:288] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:30.422116 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f7285fe2dc4c5ab0520c39ae994793" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:30.422376 13121 raft_consensus.cc:2454] T 28f7285fe2dc4c5ab0520c39ae994793 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:30.422485 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f7285fe2dc4c5ab0520c39ae994793" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:30.422724 12861 raft_consensus.cc:2454] T 28f7285fe2dc4c5ab0520c39ae994793 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:30.422726 12925 leader_election.cc:302] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:30.423012 17064 raft_consensus.cc:2781] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:30.423115 17064 raft_consensus.cc:482] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:30.423223 17064 raft_consensus.cc:3037] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:30.426079 17064 raft_consensus.cc:504] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.426734 17064 leader_election.cc:288] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:30.427076 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f7285fe2dc4c5ab0520c39ae994793" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:30.427230 13119 raft_consensus.cc:3037] T 28f7285fe2dc4c5ab0520c39ae994793 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:30.429716 13119 raft_consensus.cc:2454] T 28f7285fe2dc4c5ab0520c39ae994793 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:30.430086 12925 leader_election.cc:302] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:30.430133 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28f7285fe2dc4c5ab0520c39ae994793" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:30.430280 12861 raft_consensus.cc:3037] T 28f7285fe2dc4c5ab0520c39ae994793 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:30.430565 17064 raft_consensus.cc:2781] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:30.431279 17064 raft_consensus.cc:686] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:30.431524 17064 consensus_queue.cc:237] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:30.432608 12861 raft_consensus.cc:2454] T 28f7285fe2dc4c5ab0520c39ae994793 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:30.435635 12588 catalog_manager.cc:5484] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:30.520475 12588 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:30.528877 12730 raft_consensus.cc:1261] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:30.528926 12861 raft_consensus.cc:1261] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:30.529340 17369 consensus_queue.cc:1035] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:30.530014 17369 consensus_queue.cc:1035] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:30.535178 17534 tablet.cc:1722] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:30.536173 17534 tablet.cc:1620] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:30.536751 12579 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac (table rebalancer_dynamic_table [id=78cc96594e3b4757b71f3cd47a54fd51]). Expected version 1 got 0
I20240502 08:18:30.538724 17535 tablet.cc:1722] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:30.540535 17535 tablet.cc:1620] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:30.541883 12579 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac (table rebalancer_dynamic_table [id=78cc96594e3b4757b71f3cd47a54fd51]). Expected version 1 got 0
I20240502 08:18:30.549249 12584 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac (table rebalancer_dynamic_table [id=78cc96594e3b4757b71f3cd47a54fd51]). Expected version 1 got 0
I20240502 08:18:30.550915 17532 mvcc.cc:205] Tried to move back new op lower bound from 7023156881518436352 to 7023156880961183744. Current Snapshot: MvccSnapshot[applied={T|T < 7023156881518436352}]
I20240502 08:18:30.551550 12588 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=78cc96594e3b4757b71f3cd47a54fd51] alter complete (version 1)
I20240502 08:18:30.552176 17538 tablet.cc:1722] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:30.552597 17538 tablet.cc:1620] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:30.654105 12588 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:30.662870 17534 tablet.cc:1722] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:30.663693 17534 tablet.cc:1620] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: MemRowSet was empty: no flush needed.
I20240502 08:18:30.663754 17538 tablet.cc:1722] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:30.664175 17538 tablet.cc:1620] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:30.664477 17535 tablet.cc:1722] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:30.664880 17535 tablet.cc:1620] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:30.674324 12588 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=78cc96594e3b4757b71f3cd47a54fd51] alter complete (version 2)
I20240502 08:18:30.735375 15488 raft_consensus.cc:3037] T 24c29ddfac024726ad9ee74d77c115f9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:30.739043 17423 consensus_queue.cc:1035] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:30.739836 17520 consensus_queue.cc:1035] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:30.762892 17457 consensus_queue.cc:1035] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:30.775818 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:30.788872 17534 tablet.cc:1722] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:30.794080 17275 consensus_queue.cc:1035] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:30.799147 17538 tablet.cc:1722] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:30.799469 17535 tablet.cc:1722] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:30.800655 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=78cc96594e3b4757b71f3cd47a54fd51] alter complete (version 3)
I20240502 08:18:30.904471 12584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:30.904675 12584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:30.907974 12584 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac
I20240502 08:18:30.908643 12709 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:30 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:30.908828 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:30 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:30.909005 13101 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:30 UTC) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:30.909143 17550 tablet_replica.cc:330] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:30.909343 17550 raft_consensus.cc:2229] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:30.909570 17550 raft_consensus.cc:2258] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:30.909853 17552 tablet_replica.cc:330] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:30.910055 17552 raft_consensus.cc:2229] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:30.910408 17552 raft_consensus.cc:2258] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:30.910975 17551 tablet_replica.cc:330] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:30.911202 17551 raft_consensus.cc:2229] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:30.911408 17552 ts_tablet_manager.cc:1875] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:30.911427 17551 raft_consensus.cc:2258] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:30.911604 17550 ts_tablet_manager.cc:1875] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:30.912294 17551 ts_tablet_manager.cc:1875] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:30.917070 17551 ts_tablet_manager.cc:1888] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:30.917196 17551 log.cc:1196] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/5c15a5b0a3ac4ae7bebc65b7a371dfac
I20240502 08:18:30.917476 17550 ts_tablet_manager.cc:1888] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:30.917507 17552 ts_tablet_manager.cc:1888] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:30.917562 17551 ts_tablet_manager.cc:1909] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:30.917608 17552 log.cc:1196] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/5c15a5b0a3ac4ae7bebc65b7a371dfac
I20240502 08:18:30.917619 17550 log.cc:1196] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/5c15a5b0a3ac4ae7bebc65b7a371dfac
I20240502 08:18:30.917945 17550 ts_tablet_manager.cc:1909] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:30.918022 17552 ts_tablet_manager.cc:1909] T 5c15a5b0a3ac4ae7bebc65b7a371dfac P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting consensus metadata
I20240502 08:18:30.918548 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac (table rebalancer_dynamic_table_new_name [id=78cc96594e3b4757b71f3cd47a54fd51]) successfully deleted
I20240502 08:18:30.918927 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac (table rebalancer_dynamic_table_new_name [id=78cc96594e3b4757b71f3cd47a54fd51]) successfully deleted
I20240502 08:18:30.919137 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 5c15a5b0a3ac4ae7bebc65b7a371dfac (table rebalancer_dynamic_table_new_name [id=78cc96594e3b4757b71f3cd47a54fd51]) successfully deleted
I20240502 08:18:30.921919 17492 consensus_queue.cc:1035] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:30.928989 17492 consensus_queue.cc:1035] T 28f7285fe2dc4c5ab0520c39ae994793 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:30.996827 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:31.005406 13101 tablet_service.cc:1467] Processing CreateTablet for tablet ea85ef42ac714074a10f2a5ece4ddf8e (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=97336a3aa550477cb0f32e30f0e2a47c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.005615 15733 tablet_service.cc:1467] Processing CreateTablet for tablet ea85ef42ac714074a10f2a5ece4ddf8e (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=97336a3aa550477cb0f32e30f0e2a47c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.005978 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea85ef42ac714074a10f2a5ece4ddf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.006031 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea85ef42ac714074a10f2a5ece4ddf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.006923 15866 tablet_service.cc:1467] Processing CreateTablet for tablet c8b77a0b50464cff9c1879d57fc06ed9 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=97336a3aa550477cb0f32e30f0e2a47c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.007342 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b77a0b50464cff9c1879d57fc06ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.011322 15467 tablet_service.cc:1467] Processing CreateTablet for tablet e141ca70856946a1a3f8e21b77548736 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=97336a3aa550477cb0f32e30f0e2a47c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.011798 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e141ca70856946a1a3f8e21b77548736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.011710 15468 tablet_service.cc:1467] Processing CreateTablet for tablet c8b77a0b50464cff9c1879d57fc06ed9 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=97336a3aa550477cb0f32e30f0e2a47c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.011917 15601 tablet_service.cc:1467] Processing CreateTablet for tablet c8b77a0b50464cff9c1879d57fc06ed9 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=97336a3aa550477cb0f32e30f0e2a47c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.012126 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b77a0b50464cff9c1879d57fc06ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.012346 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8b77a0b50464cff9c1879d57fc06ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.012545 15599 tablet_service.cc:1467] Processing CreateTablet for tablet ea85ef42ac714074a10f2a5ece4ddf8e (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=97336a3aa550477cb0f32e30f0e2a47c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.012951 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea85ef42ac714074a10f2a5ece4ddf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.014870 15735 tablet_service.cc:1467] Processing CreateTablet for tablet e141ca70856946a1a3f8e21b77548736 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=97336a3aa550477cb0f32e30f0e2a47c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.015296 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e141ca70856946a1a3f8e21b77548736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.020344 13102 tablet_service.cc:1467] Processing CreateTablet for tablet e141ca70856946a1a3f8e21b77548736 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=97336a3aa550477cb0f32e30f0e2a47c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.020792 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e141ca70856946a1a3f8e21b77548736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.021579 17557 tablet_bootstrap.cc:492] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:31.023109 17557 tablet_bootstrap.cc:654] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.029363 17558 tablet_bootstrap.cc:492] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:31.030238 17560 tablet_bootstrap.cc:492] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:31.033066 17560 tablet_bootstrap.cc:654] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.033370 17558 tablet_bootstrap.cc:654] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.035295 17561 tablet_bootstrap.cc:492] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:31.037664 17561 tablet_bootstrap.cc:654] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.039343 17557 tablet_bootstrap.cc:492] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:31.039501 17557 ts_tablet_manager.cc:1367] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.005s
I20240502 08:18:31.042104 17557 raft_consensus.cc:348] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:31.042937 17557 raft_consensus.cc:374] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.043210 17557 raft_consensus.cc:729] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.043659 17557 consensus_queue.cc:260] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:31.044539 17560 tablet_bootstrap.cc:492] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:31.044706 17560 ts_tablet_manager.cc:1367] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 08:18:31.046011 17557 ts_tablet_manager.cc:1398] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240502 08:18:31.046825 17557 tablet_bootstrap.cc:492] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:31.045482 17560 raft_consensus.cc:348] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:31.048942 17561 tablet_bootstrap.cc:492] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:31.049130 17561 ts_tablet_manager.cc:1367] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240502 08:18:31.049962 17561 raft_consensus.cc:348] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:31.050251 17561 raft_consensus.cc:374] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.050702 17561 raft_consensus.cc:729] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.051623 17561 consensus_queue.cc:260] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:31.052385 17561 ts_tablet_manager.cc:1398] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:31.048326 17560 raft_consensus.cc:374] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.052592 17560 raft_consensus.cc:729] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.052968 17560 consensus_queue.cc:260] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:31.054646 17557 tablet_bootstrap.cc:654] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.054802 17560 ts_tablet_manager.cc:1398] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:18:31.055138 17560 tablet_bootstrap.cc:492] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:31.056782 17560 tablet_bootstrap.cc:654] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.060130 17558 tablet_bootstrap.cc:492] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:31.060391 17558 ts_tablet_manager.cc:1367] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.016s
I20240502 08:18:31.061196 17558 raft_consensus.cc:348] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:31.062884 17558 raft_consensus.cc:374] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.063076 17558 raft_consensus.cc:729] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.063365 17558 consensus_queue.cc:260] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:31.065773 17398 raft_consensus.cc:482] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:31.065939 17398 raft_consensus.cc:504] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:31.066588 17398 leader_election.cc:288] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:31.067080 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b77a0b50464cff9c1879d57fc06ed9" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
W20240502 08:18:31.067653 15554 leader_election.cc:341] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:31.067929 17560 tablet_bootstrap.cc:492] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:31.068085 17560 ts_tablet_manager.cc:1367] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240502 08:18:31.069150 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b77a0b50464cff9c1879d57fc06ed9" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:31.069424 12579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240502 08:18:31.069763 15555 leader_election.cc:341] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:31.069869 15555 leader_election.cc:302] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:31.070138 17560 raft_consensus.cc:348] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:31.070308 17398 raft_consensus.cc:2726] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:31.070362 17560 raft_consensus.cc:374] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.070426 17560 raft_consensus.cc:729] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.070668 17560 consensus_queue.cc:260] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:31.071179 17560 ts_tablet_manager.cc:1398] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:31.073885 17558 ts_tablet_manager.cc:1398] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20240502 08:18:31.074126 17558 tablet_bootstrap.cc:492] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:31.075668 17558 tablet_bootstrap.cc:654] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.077457 17557 tablet_bootstrap.cc:492] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:31.077790 17557 ts_tablet_manager.cc:1367] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.008s
I20240502 08:18:31.083220 17561 tablet_bootstrap.cc:492] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:31.083540 17557 raft_consensus.cc:348] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:31.083882 17557 raft_consensus.cc:374] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.084012 17557 raft_consensus.cc:729] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.084828 17561 tablet_bootstrap.cc:654] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.085567 17557 consensus_queue.cc:260] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:31.089156 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 4e008c6bc5364b488082db7a53e32526 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=be192206212c4a45827e78cf58c8000a]), partition=
I20240502 08:18:31.089534 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e008c6bc5364b488082db7a53e32526. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.089653 17557 ts_tablet_manager.cc:1398] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 08:18:31.098420 17557 tablet_bootstrap.cc:492] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:31.098994 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 4e008c6bc5364b488082db7a53e32526 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=be192206212c4a45827e78cf58c8000a]), partition=
I20240502 08:18:31.099097 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 4e008c6bc5364b488082db7a53e32526 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=be192206212c4a45827e78cf58c8000a]), partition=
I20240502 08:18:31.099303 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e008c6bc5364b488082db7a53e32526. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.099510 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e008c6bc5364b488082db7a53e32526. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.102249 17557 tablet_bootstrap.cc:654] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.111740 17570 tablet_bootstrap.cc:492] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:31.117056 17561 tablet_bootstrap.cc:492] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:31.117233 17561 ts_tablet_manager.cc:1367] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.000s
I20240502 08:18:31.118026 17561 raft_consensus.cc:348] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:31.118270 17561 raft_consensus.cc:374] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.118345 17561 raft_consensus.cc:729] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.118759 17561 consensus_queue.cc:260] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:31.119465 17561 ts_tablet_manager.cc:1398] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:31.120483 17556 tablet_bootstrap.cc:492] T c8b77a0b50464cff9c1879d57fc06ed9 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:31.120693 17558 tablet_bootstrap.cc:492] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:31.120714 17557 tablet_bootstrap.cc:492] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:31.120838 17557 ts_tablet_manager.cc:1367] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.006s
I20240502 08:18:31.120888 17558 ts_tablet_manager.cc:1367] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.047s user 0.021s sys 0.008s
I20240502 08:18:31.121655 17557 raft_consensus.cc:348] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:31.121728 17558 raft_consensus.cc:348] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:31.121891 17557 raft_consensus.cc:374] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.121948 17558 raft_consensus.cc:374] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.122026 17557 raft_consensus.cc:729] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.122064 17558 raft_consensus.cc:729] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.122282 17557 consensus_queue.cc:260] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:31.122332 17558 consensus_queue.cc:260] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:31.122498 17556 tablet_bootstrap.cc:654] T c8b77a0b50464cff9c1879d57fc06ed9 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.122880 17557 ts_tablet_manager.cc:1398] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:31.124143 17570 tablet_bootstrap.cc:654] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.127005 17558 ts_tablet_manager.cc:1398] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 08:18:31.131194 17558 tablet_bootstrap.cc:492] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:31.132447 17558 tablet_bootstrap.cc:654] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.134246 17558 tablet_bootstrap.cc:492] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:31.134421 17558 ts_tablet_manager.cc:1367] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 08:18:31.135357 17558 raft_consensus.cc:348] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:31.135613 17558 raft_consensus.cc:374] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.135725 17558 raft_consensus.cc:729] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.135975 17558 consensus_queue.cc:260] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:31.136420 17556 tablet_bootstrap.cc:492] T c8b77a0b50464cff9c1879d57fc06ed9 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:31.136628 17558 ts_tablet_manager.cc:1398] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:18:31.136646 17556 ts_tablet_manager.cc:1367] T c8b77a0b50464cff9c1879d57fc06ed9 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20240502 08:18:31.137897 17556 raft_consensus.cc:348] T c8b77a0b50464cff9c1879d57fc06ed9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:31.138947 17556 raft_consensus.cc:374] T c8b77a0b50464cff9c1879d57fc06ed9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.139123 17556 raft_consensus.cc:729] T c8b77a0b50464cff9c1879d57fc06ed9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.139427 17556 consensus_queue.cc:260] T c8b77a0b50464cff9c1879d57fc06ed9 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:31.143667 17570 tablet_bootstrap.cc:492] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:31.143858 17570 ts_tablet_manager.cc:1367] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.004s
I20240502 08:18:31.143885 17556 ts_tablet_manager.cc:1398] T c8b77a0b50464cff9c1879d57fc06ed9 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 08:18:31.144651 17570 raft_consensus.cc:348] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:31.144879 17570 raft_consensus.cc:374] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.144974 17570 raft_consensus.cc:729] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.145200 17570 consensus_queue.cc:260] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:31.145890 17570 ts_tablet_manager.cc:1398] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:31.151717 17275 raft_consensus.cc:482] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:31.151876 17275 raft_consensus.cc:504] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:31.152674 17275 leader_election.cc:288] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:31.153136 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea85ef42ac714074a10f2a5ece4ddf8e" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:31.153352 15620 raft_consensus.cc:2454] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:31.153375 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea85ef42ac714074a10f2a5ece4ddf8e" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:31.153596 13121 raft_consensus.cc:2454] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:31.153820 15690 leader_election.cc:302] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:31.154328 17275 raft_consensus.cc:2781] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:31.154443 17275 raft_consensus.cc:482] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:31.154963 17275 raft_consensus.cc:3037] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.157229 17275 raft_consensus.cc:504] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:31.158041 17275 leader_election.cc:288] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:31.158063 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea85ef42ac714074a10f2a5ece4ddf8e" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:31.158123 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea85ef42ac714074a10f2a5ece4ddf8e" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:31.158234 13121 raft_consensus.cc:3037] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.158304 15620 raft_consensus.cc:3037] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.160177 13121 raft_consensus.cc:2454] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:31.160544 15687 leader_election.cc:302] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:31.160946 17275 raft_consensus.cc:2781] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:31.161005 15620 raft_consensus.cc:2454] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:31.161134 17275 raft_consensus.cc:686] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:31.161624 17275 consensus_queue.cc:237] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:31.165701 12579 catalog_manager.cc:5484] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:31.174054 17549 raft_consensus.cc:482] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:31.174201 17549 raft_consensus.cc:504] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:31.174906 17549 leader_election.cc:288] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:31.175266 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e141ca70856946a1a3f8e21b77548736" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:31.175274 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e141ca70856946a1a3f8e21b77548736" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:31.175470 15755 raft_consensus.cc:2454] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:31.175491 15486 raft_consensus.cc:2454] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:31.175863 13056 leader_election.cc:302] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:31.176115 17549 raft_consensus.cc:2781] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:31.176239 17549 raft_consensus.cc:482] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:31.176323 17549 raft_consensus.cc:3037] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.178637 17549 raft_consensus.cc:504] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:31.179219 17549 leader_election.cc:288] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:31.179589 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e141ca70856946a1a3f8e21b77548736" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:31.179575 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e141ca70856946a1a3f8e21b77548736" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:31.179769 15755 raft_consensus.cc:3037] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.179800 15486 raft_consensus.cc:3037] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.182441 15486 raft_consensus.cc:2454] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:31.182502 15755 raft_consensus.cc:2454] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:31.182847 13055 leader_election.cc:302] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:31.183096 17549 raft_consensus.cc:2781] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:31.183238 17549 raft_consensus.cc:686] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:18:31.183511 17549 consensus_queue.cc:237] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:31.185379 17315 raft_consensus.cc:482] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:31.185528 17315 raft_consensus.cc:504] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:31.186147 17315 leader_election.cc:288] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:31.186882 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b77a0b50464cff9c1879d57fc06ed9" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:31.187080 15620 raft_consensus.cc:2454] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:31.187074 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b77a0b50464cff9c1879d57fc06ed9" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:31.187300 15886 raft_consensus.cc:2454] T c8b77a0b50464cff9c1879d57fc06ed9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:31.187551 15421 leader_election.cc:302] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:31.188091 17315 raft_consensus.cc:2781] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:31.188210 17315 raft_consensus.cc:482] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:31.188316 17315 raft_consensus.cc:3037] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.188669 12584 catalog_manager.cc:5484] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:31.190677 17315 raft_consensus.cc:504] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:31.191520 15620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b77a0b50464cff9c1879d57fc06ed9" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:31.191690 15620 raft_consensus.cc:3037] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.191967 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b77a0b50464cff9c1879d57fc06ed9" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:31.192167 15886 raft_consensus.cc:3037] T c8b77a0b50464cff9c1879d57fc06ed9 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.194751 15620 raft_consensus.cc:2454] T c8b77a0b50464cff9c1879d57fc06ed9 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:31.196609 15421 leader_election.cc:302] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:31.196857 17227 raft_consensus.cc:2781] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:31.196987 15886 raft_consensus.cc:2454] T c8b77a0b50464cff9c1879d57fc06ed9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:31.197289 17227 raft_consensus.cc:686] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:31.197544 17227 consensus_queue.cc:237] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:31.198556 17315 leader_election.cc:288] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:31.202389 12579 catalog_manager.cc:5484] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:31.210878 17576 raft_consensus.cc:482] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:31.211025 17576 raft_consensus.cc:504] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:31.211838 17576 leader_election.cc:288] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:31.212318 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e008c6bc5364b488082db7a53e32526" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:31.212332 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e008c6bc5364b488082db7a53e32526" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:31.212574 15755 raft_consensus.cc:2454] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:31.212566 12992 raft_consensus.cc:2454] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:31.212915 15418 leader_election.cc:302] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:31.213573 17576 raft_consensus.cc:2781] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:31.213686 17576 raft_consensus.cc:482] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:31.213766 17576 raft_consensus.cc:3037] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.215773 17576 raft_consensus.cc:504] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:31.216233 17576 leader_election.cc:288] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:31.216660 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e008c6bc5364b488082db7a53e32526" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:31.216632 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e008c6bc5364b488082db7a53e32526" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:31.216849 12992 raft_consensus.cc:3037] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.216987 15755 raft_consensus.cc:3037] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.219543 12992 raft_consensus.cc:2454] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:31.219844 15755 raft_consensus.cc:2454] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:31.219913 15419 leader_election.cc:302] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:31.220553 17576 raft_consensus.cc:2781] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:31.220687 17576 raft_consensus.cc:686] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:31.220984 17576 consensus_queue.cc:237] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:31.229288 12579 catalog_manager.cc:5484] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:31.354702 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:31.361063 12992 raft_consensus.cc:1261] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:31.361126 15754 raft_consensus.cc:1261] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:31.361688 17577 consensus_queue.cc:1035] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:31.362114 17577 consensus_queue.cc:1035] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:31.367579 12579 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet 4e008c6bc5364b488082db7a53e32526 (table rebalancer_dynamic_table [id=be192206212c4a45827e78cf58c8000a]). Expected version 1 got 0
I20240502 08:18:31.369115 12584 catalog_manager.cc:5576] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757) does not have the latest schema for tablet 4e008c6bc5364b488082db7a53e32526 (table rebalancer_dynamic_table [id=be192206212c4a45827e78cf58c8000a]). Expected version 1 got 0
I20240502 08:18:31.375121 17585 tablet.cc:1722] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:31.375734 17585 tablet.cc:1620] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:31.375772 17586 tablet.cc:1722] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:31.378522 17586 tablet.cc:1620] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:31.379698 17587 tablet.cc:1722] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:31.381773 17587 tablet.cc:1620] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:31.381980 12584 catalog_manager.cc:5576] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851) does not have the latest schema for tablet 4e008c6bc5364b488082db7a53e32526 (table rebalancer_dynamic_table [id=be192206212c4a45827e78cf58c8000a]). Expected version 1 got 0
I20240502 08:18:31.384439 12579 catalog_manager.cc:5576] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757) does not have the latest schema for tablet 4e008c6bc5364b488082db7a53e32526 (table rebalancer_dynamic_table [id=be192206212c4a45827e78cf58c8000a]). Expected version 1 got 0
I20240502 08:18:31.390296 12585 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=be192206212c4a45827e78cf58c8000a] alter complete (version 1)
I20240502 08:18:31.492058 12579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:31.499994 17585 tablet.cc:1722] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:31.500411 17585 tablet.cc:1620] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: MemRowSet was empty: no flush needed.
I20240502 08:18:31.500330 17586 tablet.cc:1722] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:31.500774 17586 tablet.cc:1620] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:31.504268 17587 tablet.cc:1722] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:31.504716 17587 tablet.cc:1620] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:31.510310 12579 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=be192206212c4a45827e78cf58c8000a] alter complete (version 2)
I20240502 08:18:31.554563 17275 consensus_queue.cc:1035] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:31.560098 17275 consensus_queue.cc:1035] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:31.566397 17549 consensus_queue.cc:1035] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:31.571188 17369 consensus_queue.cc:1035] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:31.615514 12588 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:31.623116 17585 tablet.cc:1722] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:31.627388 17587 tablet.cc:1722] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:31.630688 17586 tablet.cc:1722] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:31.633931 12588 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=be192206212c4a45827e78cf58c8000a] alter complete (version 3)
I20240502 08:18:31.700680 17577 consensus_queue.cc:1035] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:31.704824 17227 consensus_queue.cc:1035] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:31.726713 12588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:31.735949 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 5be106986ad247c597666d46a1dede7d (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=107c7945253947cda94a688a2c9373cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.736210 12709 tablet_service.cc:1467] Processing CreateTablet for tablet 5be106986ad247c597666d46a1dede7d (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=107c7945253947cda94a688a2c9373cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.736442 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be106986ad247c597666d46a1dede7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.736603 12709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be106986ad247c597666d46a1dede7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.737774 12588 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:31.737902 12588 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:31.738845 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 5be106986ad247c597666d46a1dede7d (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=107c7945253947cda94a688a2c9373cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.739202 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be106986ad247c597666d46a1dede7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.740285 12588 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 4e008c6bc5364b488082db7a53e32526
I20240502 08:18:31.741003 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e008c6bc5364b488082db7a53e32526 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:31 UTC) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:31.740983 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e008c6bc5364b488082db7a53e32526 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:31 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:31.745383 13101 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4a5f0c5c184b7f975f7bcc59cb6b4c (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=107c7945253947cda94a688a2c9373cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.745558 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4a5f0c5c184b7f975f7bcc59cb6b4c (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=107c7945253947cda94a688a2c9373cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.745762 13101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d4a5f0c5c184b7f975f7bcc59cb6b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.745980 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d4a5f0c5c184b7f975f7bcc59cb6b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.746742 15866 tablet_service.cc:1467] Processing CreateTablet for tablet f0ab46265cca4e8e9775abedbe8affda (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=107c7945253947cda94a688a2c9373cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.747133 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0ab46265cca4e8e9775abedbe8affda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.747303 12708 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4a5f0c5c184b7f975f7bcc59cb6b4c (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=107c7945253947cda94a688a2c9373cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.747669 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d4a5f0c5c184b7f975f7bcc59cb6b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.750595 12972 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e008c6bc5364b488082db7a53e32526 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:31 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:31.751391 12841 tablet_service.cc:1467] Processing CreateTablet for tablet f0ab46265cca4e8e9775abedbe8affda (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=107c7945253947cda94a688a2c9373cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.751753 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0ab46265cca4e8e9775abedbe8affda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.752298 12710 tablet_service.cc:1467] Processing CreateTablet for tablet f0ab46265cca4e8e9775abedbe8affda (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=107c7945253947cda94a688a2c9373cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:31.752653 12710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0ab46265cca4e8e9775abedbe8affda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.768760 17601 tablet_replica.cc:330] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:31.769827 17601 raft_consensus.cc:2229] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:31.770642 17601 raft_consensus.cc:2258] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:31.771643 17602 tablet_bootstrap.cc:492] T f0ab46265cca4e8e9775abedbe8affda P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:31.773216 17600 tablet_bootstrap.cc:492] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:31.774044 17602 tablet_bootstrap.cc:654] T f0ab46265cca4e8e9775abedbe8affda P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.773959 17601 ts_tablet_manager.cc:1875] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:31.779901 17604 tablet_bootstrap.cc:492] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:31.781787 17604 tablet_bootstrap.cc:654] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.783017 17603 tablet_bootstrap.cc:492] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:31.783063 17600 tablet_bootstrap.cc:654] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.784412 17603 tablet_bootstrap.cc:654] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.785647 17605 tablet_bootstrap.cc:492] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:31.786269 17599 tablet_replica.cc:330] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:31.786780 17606 tablet_bootstrap.cc:492] T f0ab46265cca4e8e9775abedbe8affda P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:31.787705 17603 tablet_bootstrap.cc:492] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:31.787989 17599 raft_consensus.cc:2229] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:31.788295 17606 tablet_bootstrap.cc:654] T f0ab46265cca4e8e9775abedbe8affda P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.788951 17603 ts_tablet_manager.cc:1367] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:18:31.790491 17599 raft_consensus.cc:2258] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:31.791904 17605 tablet_bootstrap.cc:654] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.791920 17602 tablet_bootstrap.cc:492] T f0ab46265cca4e8e9775abedbe8affda P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:31.792047 17602 ts_tablet_manager.cc:1367] T f0ab46265cca4e8e9775abedbe8affda P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.008s
I20240502 08:18:31.792802 17599 ts_tablet_manager.cc:1875] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:31.792779 17602 raft_consensus.cc:348] T f0ab46265cca4e8e9775abedbe8affda P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.792940 17603 raft_consensus.cc:348] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.793140 17602 raft_consensus.cc:374] T f0ab46265cca4e8e9775abedbe8affda P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.793243 17602 raft_consensus.cc:729] T f0ab46265cca4e8e9775abedbe8affda P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.793401 17603 raft_consensus.cc:374] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.793466 17602 consensus_queue.cc:260] T f0ab46265cca4e8e9775abedbe8affda P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.793716 17603 raft_consensus.cc:729] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.796468 17602 ts_tablet_manager.cc:1398] T f0ab46265cca4e8e9775abedbe8affda P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:31.797308 17603 consensus_queue.cc:260] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.798103 17601 ts_tablet_manager.cc:1888] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:31.799759 17603 ts_tablet_manager.cc:1398] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 08:18:31.800448 17604 tablet_bootstrap.cc:492] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:31.800587 17604 ts_tablet_manager.cc:1367] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240502 08:18:31.801182 17604 raft_consensus.cc:348] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.801360 17604 raft_consensus.cc:374] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.801462 17604 raft_consensus.cc:729] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.801697 17604 consensus_queue.cc:260] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.803099 17604 ts_tablet_manager.cc:1398] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:31.803411 17604 tablet_bootstrap.cc:492] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:31.806186 17604 tablet_bootstrap.cc:654] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.811615 17603 tablet_bootstrap.cc:492] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:31.811995 17606 tablet_bootstrap.cc:492] T f0ab46265cca4e8e9775abedbe8affda P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:31.812182 17606 ts_tablet_manager.cc:1367] T f0ab46265cca4e8e9775abedbe8affda P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.006s
I20240502 08:18:31.812978 17606 raft_consensus.cc:348] T f0ab46265cca4e8e9775abedbe8affda P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.813222 17606 raft_consensus.cc:374] T f0ab46265cca4e8e9775abedbe8affda P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.813314 17606 raft_consensus.cc:729] T f0ab46265cca4e8e9775abedbe8affda P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.813529 17606 consensus_queue.cc:260] T f0ab46265cca4e8e9775abedbe8affda P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.814407 17601 log.cc:1196] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/4e008c6bc5364b488082db7a53e32526
I20240502 08:18:31.814880 17598 tablet_replica.cc:330] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:31.815086 17598 raft_consensus.cc:2229] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:31.815217 17601 ts_tablet_manager.cc:1909] T 4e008c6bc5364b488082db7a53e32526 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:31.815300 17598 raft_consensus.cc:2258] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:31.816437 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 4e008c6bc5364b488082db7a53e32526 (table rebalancer_dynamic_table_new_name [id=be192206212c4a45827e78cf58c8000a]) successfully deleted
I20240502 08:18:31.817479 17603 tablet_bootstrap.cc:654] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.819002 17598 ts_tablet_manager.cc:1875] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:31.819602 17606 ts_tablet_manager.cc:1398] T f0ab46265cca4e8e9775abedbe8affda P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 08:18:31.820752 17605 tablet_bootstrap.cc:492] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:31.820971 17605 ts_tablet_manager.cc:1367] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.000s
I20240502 08:18:31.821890 17605 raft_consensus.cc:348] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.822189 17605 raft_consensus.cc:374] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.822362 17605 raft_consensus.cc:729] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.822723 17605 consensus_queue.cc:260] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.823727 17605 ts_tablet_manager.cc:1398] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:31.833038 17599 ts_tablet_manager.cc:1888] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:31.833169 17599 log.cc:1196] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/4e008c6bc5364b488082db7a53e32526
I20240502 08:18:31.833580 17599 ts_tablet_manager.cc:1909] T 4e008c6bc5364b488082db7a53e32526 P db7893bb29a047b3b82cc60bfd8930a3: Deleting consensus metadata
I20240502 08:18:31.834667 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 4e008c6bc5364b488082db7a53e32526 (table rebalancer_dynamic_table_new_name [id=be192206212c4a45827e78cf58c8000a]) successfully deleted
I20240502 08:18:31.828601 17598 ts_tablet_manager.cc:1888] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:31.842629 17598 log.cc:1196] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/4e008c6bc5364b488082db7a53e32526
I20240502 08:18:31.843403 17598 ts_tablet_manager.cc:1909] T 4e008c6bc5364b488082db7a53e32526 P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:31.845103 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 4e008c6bc5364b488082db7a53e32526 (table rebalancer_dynamic_table_new_name [id=be192206212c4a45827e78cf58c8000a]) successfully deleted
I20240502 08:18:31.848047 17604 tablet_bootstrap.cc:492] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:31.848238 17604 ts_tablet_manager.cc:1367] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.007s
I20240502 08:18:31.849073 17604 raft_consensus.cc:348] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.849344 17604 raft_consensus.cc:374] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.849422 17604 raft_consensus.cc:729] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.849700 17604 consensus_queue.cc:260] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.850286 17604 ts_tablet_manager.cc:1398] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:31.851594 17603 tablet_bootstrap.cc:492] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:31.851819 17603 ts_tablet_manager.cc:1367] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.010s
I20240502 08:18:31.852674 17600 tablet_bootstrap.cc:492] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:31.852874 17600 ts_tablet_manager.cc:1367] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.080s user 0.006s sys 0.003s
I20240502 08:18:31.853055 17603 raft_consensus.cc:348] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.853329 17603 raft_consensus.cc:374] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.853453 17603 raft_consensus.cc:729] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.853750 17603 consensus_queue.cc:260] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.853809 17600 raft_consensus.cc:348] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.854100 17600 raft_consensus.cc:374] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.854375 17600 raft_consensus.cc:729] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.854676 17600 consensus_queue.cc:260] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.856503 17603 ts_tablet_manager.cc:1398] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 08:18:31.856812 17603 tablet_bootstrap.cc:492] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:31.857285 17600 ts_tablet_manager.cc:1398] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:31.858212 17603 tablet_bootstrap.cc:654] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.858032 12585 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:31.859925 17398 raft_consensus.cc:482] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:31.860118 17398 raft_consensus.cc:504] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.860754 17398 leader_election.cc:288] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:31.861480 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4a5f0c5c184b7f975f7bcc59cb6b4c" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:31.861688 12727 raft_consensus.cc:2454] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:31.862990 15554 leader_election.cc:302] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:31.864466 17398 raft_consensus.cc:2781] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:31.864568 17398 raft_consensus.cc:482] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:31.864668 17398 raft_consensus.cc:3037] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.866833 17398 raft_consensus.cc:504] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.867316 17398 leader_election.cc:288] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:31.868175 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4a5f0c5c184b7f975f7bcc59cb6b4c" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:31.868221 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4a5f0c5c184b7f975f7bcc59cb6b4c" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:31.868407 12727 raft_consensus.cc:3037] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.868424 13121 raft_consensus.cc:2454] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:31.871088 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4a5f0c5c184b7f975f7bcc59cb6b4c" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:31.871286 13121 raft_consensus.cc:3037] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.874588 13121 raft_consensus.cc:2454] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:31.875873 15553 leader_election.cc:302] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:31.876160 17398 raft_consensus.cc:2781] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:31.876365 17398 raft_consensus.cc:686] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:31.876694 17398 consensus_queue.cc:237] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.877039 12727 raft_consensus.cc:2454] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:31.877895 17154 raft_consensus.cc:482] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:31.879346 17154 raft_consensus.cc:504] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.880033 17154 leader_election.cc:288] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:31.880892 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ab46265cca4e8e9775abedbe8affda" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:31.881129 15886 raft_consensus.cc:2454] T f0ab46265cca4e8e9775abedbe8affda P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:31.881829 12665 leader_election.cc:302] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 29b035434087435cbaf4ebcac98eedee; no voters:
I20240502 08:18:31.883292 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ab46265cca4e8e9775abedbe8affda" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:31.883517 12861 raft_consensus.cc:2454] T f0ab46265cca4e8e9775abedbe8affda P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 0.
I20240502 08:18:31.884446 15866 tablet_service.cc:1467] Processing CreateTablet for tablet 7388dad3024a44efa7bbe748e2ef5168 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55b9aa3e2ea442b8b075ed531f615dc0]), partition=
I20240502 08:18:31.884846 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7388dad3024a44efa7bbe748e2ef5168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.887895 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 7388dad3024a44efa7bbe748e2ef5168 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55b9aa3e2ea442b8b075ed531f615dc0]), partition=
I20240502 08:18:31.888224 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7388dad3024a44efa7bbe748e2ef5168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.888844 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 7388dad3024a44efa7bbe748e2ef5168 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55b9aa3e2ea442b8b075ed531f615dc0]), partition=
I20240502 08:18:31.889199 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7388dad3024a44efa7bbe748e2ef5168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.895387 17606 tablet_bootstrap.cc:492] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:31.896391 17512 raft_consensus.cc:2781] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:31.896497 17512 raft_consensus.cc:482] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:31.896612 17512 raft_consensus.cc:3037] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.896731 17606 tablet_bootstrap.cc:654] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.899317 17512 raft_consensus.cc:504] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.899968 17512 leader_election.cc:288] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:31.906447 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ab46265cca4e8e9775abedbe8affda" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:31.906720 15887 raft_consensus.cc:3037] T f0ab46265cca4e8e9775abedbe8affda P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.909132 17606 tablet_bootstrap.cc:492] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:31.909266 17606 ts_tablet_manager.cc:1367] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20240502 08:18:31.909667 15887 raft_consensus.cc:2454] T f0ab46265cca4e8e9775abedbe8affda P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:31.909999 17606 raft_consensus.cc:348] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:31.910209 17606 raft_consensus.cc:374] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.910477 12665 leader_election.cc:302] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 29b035434087435cbaf4ebcac98eedee; no voters:
I20240502 08:18:31.910295 17606 raft_consensus.cc:729] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.910786 17606 consensus_queue.cc:260] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:31.911088 17512 raft_consensus.cc:2781] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:31.911221 17512 raft_consensus.cc:686] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:18:31.911464 17512 consensus_queue.cc:237] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.912791 17606 ts_tablet_manager.cc:1398] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 08:18:31.913933 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0ab46265cca4e8e9775abedbe8affda" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:31.914153 12861 raft_consensus.cc:3037] T f0ab46265cca4e8e9775abedbe8affda P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:31.934973 12861 raft_consensus.cc:2454] T f0ab46265cca4e8e9775abedbe8affda P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:18:31.938196 17600 tablet_bootstrap.cc:492] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:31.938576 12584 catalog_manager.cc:5484] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:31.939713 17600 tablet_bootstrap.cc:654] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.940586 17602 tablet_bootstrap.cc:492] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:31.942144 17602 tablet_bootstrap.cc:654] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:31.949291 17603 tablet_bootstrap.cc:492] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:31.949451 17603 ts_tablet_manager.cc:1367] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.093s user 0.035s sys 0.046s
I20240502 08:18:31.950136 17603 raft_consensus.cc:348] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.950349 17603 raft_consensus.cc:374] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.950803 12588 catalog_manager.cc:5484] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:31.953111 17600 tablet_bootstrap.cc:492] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:31.954277 17600 ts_tablet_manager.cc:1367] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20240502 08:18:31.955384 17600 raft_consensus.cc:348] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:31.955655 17600 raft_consensus.cc:374] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.955755 17600 raft_consensus.cc:729] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.956008 17600 consensus_queue.cc:260] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:31.956696 17600 ts_tablet_manager.cc:1398] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:31.957494 17603 raft_consensus.cc:729] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.957814 17603 consensus_queue.cc:260] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:31.958326 17603 ts_tablet_manager.cc:1398] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
W20240502 08:18:31.961817 12664 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.5:44131, user_credentials={real_user=slave}} blocked reactor thread for 48622.9us
I20240502 08:18:31.968473 17602 tablet_bootstrap.cc:492] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:31.968675 17602 ts_tablet_manager.cc:1367] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.001s
I20240502 08:18:31.969396 17602 raft_consensus.cc:348] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:31.969611 17602 raft_consensus.cc:374] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:31.969683 17602 raft_consensus.cc:729] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:31.969887 17602 consensus_queue.cc:260] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:31.970537 17602 ts_tablet_manager.cc:1398] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:18:32.077733 17553 raft_consensus.cc:482] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:32.078300 17553 raft_consensus.cc:504] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:32.079180 17553 leader_election.cc:288] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:32.079511 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7388dad3024a44efa7bbe748e2ef5168" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:32.079638 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7388dad3024a44efa7bbe748e2ef5168" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:32.079730 12861 raft_consensus.cc:2454] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:32.079824 15887 raft_consensus.cc:2454] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:32.080143 12926 leader_election.cc:302] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:32.080456 17553 raft_consensus.cc:2781] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:32.080632 17553 raft_consensus.cc:482] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:32.080735 17553 raft_consensus.cc:3037] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.083966 17553 raft_consensus.cc:504] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:32.084926 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7388dad3024a44efa7bbe748e2ef5168" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:32.085043 17553 leader_election.cc:288] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:32.085081 12861 raft_consensus.cc:3037] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.085163 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7388dad3024a44efa7bbe748e2ef5168" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:32.085325 15887 raft_consensus.cc:3037] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.087705 12861 raft_consensus.cc:2454] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:32.087944 15887 raft_consensus.cc:2454] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:32.088438 12927 leader_election.cc:302] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 60b6e265e7754b7187307d05ef405dd4; no voters:
I20240502 08:18:32.088853 17553 raft_consensus.cc:2781] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:32.089037 17553 raft_consensus.cc:686] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:32.089370 17553 consensus_queue.cc:237] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:32.093669 12588 catalog_manager.cc:5484] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:32.115849 17553 raft_consensus.cc:482] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:32.116017 17553 raft_consensus.cc:504] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.117273 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be106986ad247c597666d46a1dede7d" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:32.117419 17553 leader_election.cc:288] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:32.117513 13120 raft_consensus.cc:2454] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:32.117658 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be106986ad247c597666d46a1dede7d" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:32.117864 12727 raft_consensus.cc:2454] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 0.
I20240502 08:18:32.117882 12925 leader_election.cc:302] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:32.118173 17553 raft_consensus.cc:2781] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:32.118265 17553 raft_consensus.cc:482] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:32.118372 17553 raft_consensus.cc:3037] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.121273 17553 raft_consensus.cc:504] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.122095 17553 leader_election.cc:288] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:32.122195 13120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be106986ad247c597666d46a1dede7d" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:32.122381 13120 raft_consensus.cc:3037] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.124504 13120 raft_consensus.cc:2454] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:32.124923 12925 leader_election.cc:302] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:32.125190 17553 raft_consensus.cc:2781] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:32.125342 17553 raft_consensus.cc:686] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:18:32.125654 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be106986ad247c597666d46a1dede7d" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:32.125613 17553 consensus_queue.cc:237] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.125828 12727 raft_consensus.cc:3037] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.128213 12727 raft_consensus.cc:2454] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:18:32.129680 12588 catalog_manager.cc:5484] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:32.206317 12588 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:32.223809 12861 raft_consensus.cc:1261] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:32.223874 15887 raft_consensus.cc:1261] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:32.224566 17492 consensus_queue.cc:1035] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:32.225296 17492 consensus_queue.cc:1035] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:32.241376 17631 tablet.cc:1722] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:32.241977 17631 tablet.cc:1620] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:32.244247 12588 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 7388dad3024a44efa7bbe748e2ef5168 (table rebalancer_dynamic_table [id=55b9aa3e2ea442b8b075ed531f615dc0]). Expected version 1 got 0
I20240502 08:18:32.256601 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=55b9aa3e2ea442b8b075ed531f615dc0] alter complete (version 1)
I20240502 08:18:32.262557 17632 tablet.cc:1722] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:32.264230 17632 tablet.cc:1620] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:32.265041 12584 catalog_manager.cc:5576] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303) does not have the latest schema for tablet 7388dad3024a44efa7bbe748e2ef5168 (table rebalancer_dynamic_table [id=55b9aa3e2ea442b8b075ed531f615dc0]). Expected version 1 got 0
I20240502 08:18:32.273267 12584 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet 7388dad3024a44efa7bbe748e2ef5168 (table rebalancer_dynamic_table [id=55b9aa3e2ea442b8b075ed531f615dc0]). Expected version 1 got 0
I20240502 08:18:32.298804 17633 tablet.cc:1722] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:32.299574 17633 tablet.cc:1620] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:32.302040 12584 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet 7388dad3024a44efa7bbe748e2ef5168 (table rebalancer_dynamic_table [id=55b9aa3e2ea442b8b075ed531f615dc0]). Expected version 1 got 0
I20240502 08:18:32.391944 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:32.399003 17631 tablet.cc:1722] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:32.399525 17631 tablet.cc:1620] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: MemRowSet was empty: no flush needed.
I20240502 08:18:32.400287 17633 tablet.cc:1722] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:32.400769 17633 tablet.cc:1620] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:32.403555 17632 tablet.cc:1722] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:32.403956 17632 tablet.cc:1620] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: MemRowSet was empty: no flush needed.
I20240502 08:18:32.409950 12588 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=55b9aa3e2ea442b8b075ed531f615dc0] alter complete (version 2)
I20240502 08:18:32.410427 17154 consensus_queue.cc:1035] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:32.416672 17154 consensus_queue.cc:1035] T f0ab46265cca4e8e9775abedbe8affda P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:32.437566 17622 consensus_queue.cc:1035] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:32.442962 17398 consensus_queue.cc:1035] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:32.512351 12584 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:32.517374 17492 consensus_queue.cc:1035] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:32.520745 17631 tablet.cc:1722] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:32.522372 17632 tablet.cc:1722] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:32.523262 17633 tablet.cc:1722] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:32.525697 17642 consensus_queue.cc:1035] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:32.535934 12584 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=55b9aa3e2ea442b8b075ed531f615dc0] alter complete (version 3)
I20240502 08:18:32.641125 12583 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:32.641358 12583 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:32.643992 12583 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet 7388dad3024a44efa7bbe748e2ef5168
I20240502 08:18:32.644853 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 7388dad3024a44efa7bbe748e2ef5168 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:32 UTC) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:32.644853 12841 tablet_service.cc:1514] Processing DeleteTablet for tablet 7388dad3024a44efa7bbe748e2ef5168 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:32 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:32.645363 17645 tablet_replica.cc:330] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:32.645560 15866 tablet_service.cc:1514] Processing DeleteTablet for tablet 7388dad3024a44efa7bbe748e2ef5168 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:32 UTC) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:32.645696 17645 raft_consensus.cc:2229] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:32.645911 17645 raft_consensus.cc:2258] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:32.646548 17644 tablet_replica.cc:330] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:32.646735 17644 raft_consensus.cc:2229] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:32.647080 17644 raft_consensus.cc:2258] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:32.647768 17644 ts_tablet_manager.cc:1875] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:32.648075 17645 ts_tablet_manager.cc:1875] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:32.652998 17644 ts_tablet_manager.cc:1888] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:32.653100 17644 log.cc:1196] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/7388dad3024a44efa7bbe748e2ef5168
I20240502 08:18:32.653436 17644 ts_tablet_manager.cc:1909] T 7388dad3024a44efa7bbe748e2ef5168 P 60b6e265e7754b7187307d05ef405dd4: Deleting consensus metadata
I20240502 08:18:32.655543 17645 ts_tablet_manager.cc:1888] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:32.655656 17645 log.cc:1196] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/7388dad3024a44efa7bbe748e2ef5168
I20240502 08:18:32.656011 17645 ts_tablet_manager.cc:1909] T 7388dad3024a44efa7bbe748e2ef5168 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:32.656819 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 7388dad3024a44efa7bbe748e2ef5168 (table rebalancer_dynamic_table_new_name [id=55b9aa3e2ea442b8b075ed531f615dc0]) successfully deleted
I20240502 08:18:32.658165 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 7388dad3024a44efa7bbe748e2ef5168 (table rebalancer_dynamic_table_new_name [id=55b9aa3e2ea442b8b075ed531f615dc0]) successfully deleted
I20240502 08:18:32.662348 17646 tablet_replica.cc:330] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:32.662593 17646 raft_consensus.cc:2229] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:32.662865 17646 raft_consensus.cc:2258] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:32.663753 17646 ts_tablet_manager.cc:1875] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:32.668044 12583 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:32.668594 17646 ts_tablet_manager.cc:1888] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:32.668735 17646 log.cc:1196] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/7388dad3024a44efa7bbe748e2ef5168
I20240502 08:18:32.669231 17646 ts_tablet_manager.cc:1909] T 7388dad3024a44efa7bbe748e2ef5168 P 10f0e36203914b85b77e78609ffced98: Deleting consensus metadata
I20240502 08:18:32.670184 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 7388dad3024a44efa7bbe748e2ef5168 (table rebalancer_dynamic_table_new_name [id=55b9aa3e2ea442b8b075ed531f615dc0]) successfully deleted
I20240502 08:18:32.678689 12841 tablet_service.cc:1467] Processing CreateTablet for tablet c65f66921e73457eb07acec0670b2426 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=94d0e1dcc3fb43e694fe94acec28e365]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:32.678689 15468 tablet_service.cc:1467] Processing CreateTablet for tablet c65f66921e73457eb07acec0670b2426 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=94d0e1dcc3fb43e694fe94acec28e365]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:32.679090 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65f66921e73457eb07acec0670b2426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.679100 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65f66921e73457eb07acec0670b2426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.684540 12970 tablet_service.cc:1467] Processing CreateTablet for tablet c65f66921e73457eb07acec0670b2426 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=94d0e1dcc3fb43e694fe94acec28e365]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:32.684932 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65f66921e73457eb07acec0670b2426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.685086 17648 tablet_bootstrap.cc:492] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:32.686161 13102 tablet_service.cc:1467] Processing CreateTablet for tablet e8e1b3aa05d146048d50ae5c38f6f56b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=94d0e1dcc3fb43e694fe94acec28e365]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:32.686599 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e1b3aa05d146048d50ae5c38f6f56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.686743 17648 tablet_bootstrap.cc:654] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:32.688189 12708 tablet_service.cc:1467] Processing CreateTablet for tablet e8e1b3aa05d146048d50ae5c38f6f56b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=94d0e1dcc3fb43e694fe94acec28e365]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:32.688616 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e1b3aa05d146048d50ae5c38f6f56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.694316 15866 tablet_service.cc:1467] Processing CreateTablet for tablet e8e1b3aa05d146048d50ae5c38f6f56b (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=94d0e1dcc3fb43e694fe94acec28e365]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:32.694753 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e1b3aa05d146048d50ae5c38f6f56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.698799 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 30001d4bd6e943f88d28894ef59cbe18 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=94d0e1dcc3fb43e694fe94acec28e365]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:32.699141 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30001d4bd6e943f88d28894ef59cbe18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.713232 17653 tablet_bootstrap.cc:492] T e8e1b3aa05d146048d50ae5c38f6f56b P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:32.715998 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 30001d4bd6e943f88d28894ef59cbe18 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=94d0e1dcc3fb43e694fe94acec28e365]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:32.717478 17651 tablet_bootstrap.cc:492] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:32.718123 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30001d4bd6e943f88d28894ef59cbe18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.722662 17653 tablet_bootstrap.cc:654] T e8e1b3aa05d146048d50ae5c38f6f56b P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:32.723306 17651 tablet_bootstrap.cc:654] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:32.736238 17652 tablet_bootstrap.cc:492] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:32.736781 17655 tablet_bootstrap.cc:492] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:32.737838 17653 tablet_bootstrap.cc:492] T e8e1b3aa05d146048d50ae5c38f6f56b P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:32.738035 17653 ts_tablet_manager.cc:1367] T e8e1b3aa05d146048d50ae5c38f6f56b P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.002s
I20240502 08:18:32.739395 17652 tablet_bootstrap.cc:654] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:32.739284 17653 raft_consensus.cc:348] T e8e1b3aa05d146048d50ae5c38f6f56b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.739585 17653 raft_consensus.cc:374] T e8e1b3aa05d146048d50ae5c38f6f56b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:32.739682 17653 raft_consensus.cc:729] T e8e1b3aa05d146048d50ae5c38f6f56b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:32.739933 17653 consensus_queue.cc:260] T e8e1b3aa05d146048d50ae5c38f6f56b P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.740983 17655 tablet_bootstrap.cc:654] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:32.741158 17653 ts_tablet_manager.cc:1398] T e8e1b3aa05d146048d50ae5c38f6f56b P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:32.741257 17648 tablet_bootstrap.cc:492] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:32.742770 17648 ts_tablet_manager.cc:1367] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.058s user 0.021s sys 0.017s
I20240502 08:18:32.743359 17652 tablet_bootstrap.cc:492] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:32.743393 17651 tablet_bootstrap.cc:492] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:32.743523 17651 ts_tablet_manager.cc:1367] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240502 08:18:32.743619 17648 raft_consensus.cc:348] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:32.743988 17648 raft_consensus.cc:374] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:32.744138 17652 ts_tablet_manager.cc:1367] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 08:18:32.744149 17648 raft_consensus.cc:729] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:32.744266 17651 raft_consensus.cc:348] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:32.744536 17651 raft_consensus.cc:374] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:32.744457 17648 consensus_queue.cc:260] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:32.744652 17651 raft_consensus.cc:729] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:32.744786 17652 raft_consensus.cc:348] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:32.744952 17651 consensus_queue.cc:260] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:32.746376 17648 ts_tablet_manager.cc:1398] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:32.746699 17648 tablet_bootstrap.cc:492] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:32.746815 17652 raft_consensus.cc:374] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:32.747179 17652 raft_consensus.cc:729] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:32.747576 17652 consensus_queue.cc:260] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:32.748970 17652 ts_tablet_manager.cc:1398] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:18:32.749855 12588 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:32.750061 17654 tablet_bootstrap.cc:492] T e8e1b3aa05d146048d50ae5c38f6f56b P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:32.753415 17654 tablet_bootstrap.cc:654] T e8e1b3aa05d146048d50ae5c38f6f56b P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:32.754534 17648 tablet_bootstrap.cc:654] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:32.745510 17651 ts_tablet_manager.cc:1398] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:32.757643 17655 tablet_bootstrap.cc:492] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:32.757845 17655 ts_tablet_manager.cc:1367] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.002s
I20240502 08:18:32.758725 17655 raft_consensus.cc:348] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:32.758985 17655 raft_consensus.cc:374] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:32.759120 17655 raft_consensus.cc:729] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:32.759466 17655 consensus_queue.cc:260] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:32.751272 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 30001d4bd6e943f88d28894ef59cbe18 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=94d0e1dcc3fb43e694fe94acec28e365]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:32.761384 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30001d4bd6e943f88d28894ef59cbe18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.761612 17655 ts_tablet_manager.cc:1398] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:18:32.761909 17655 tablet_bootstrap.cc:492] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:32.763726 17655 tablet_bootstrap.cc:654] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:32.765983 15733 tablet_service.cc:1467] Processing CreateTablet for tablet a6afd792544a45acb79669ed249559a9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2c10382fe86b483cab4494831f51bb62]), partition=
I20240502 08:18:32.766323 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6afd792544a45acb79669ed249559a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.766525 12842 tablet_service.cc:1467] Processing CreateTablet for tablet a6afd792544a45acb79669ed249559a9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2c10382fe86b483cab4494831f51bb62]), partition=
I20240502 08:18:32.766829 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6afd792544a45acb79669ed249559a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.767925 17651 tablet_bootstrap.cc:492] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:32.769548 17651 tablet_bootstrap.cc:654] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:32.774631 17648 tablet_bootstrap.cc:492] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:32.774833 17648 ts_tablet_manager.cc:1367] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20240502 08:18:32.775957 17648 raft_consensus.cc:348] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:32.776213 17648 raft_consensus.cc:374] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:32.776314 17648 raft_consensus.cc:729] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:32.776563 17648 consensus_queue.cc:260] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:32.778389 17648 ts_tablet_manager.cc:1398] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:32.770817 12708 tablet_service.cc:1467] Processing CreateTablet for tablet a6afd792544a45acb79669ed249559a9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2c10382fe86b483cab4494831f51bb62]), partition=
I20240502 08:18:32.780754 12708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6afd792544a45acb79669ed249559a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.785794 17654 tablet_bootstrap.cc:492] T e8e1b3aa05d146048d50ae5c38f6f56b P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:32.786017 17654 ts_tablet_manager.cc:1367] T e8e1b3aa05d146048d50ae5c38f6f56b P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20240502 08:18:32.786631 17654 raft_consensus.cc:348] T e8e1b3aa05d146048d50ae5c38f6f56b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.787525 17654 raft_consensus.cc:374] T e8e1b3aa05d146048d50ae5c38f6f56b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:32.787632 17654 raft_consensus.cc:729] T e8e1b3aa05d146048d50ae5c38f6f56b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:32.787719 17663 tablet_bootstrap.cc:492] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:32.789219 17654 consensus_queue.cc:260] T e8e1b3aa05d146048d50ae5c38f6f56b P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.789949 17651 tablet_bootstrap.cc:492] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:32.790133 17651 ts_tablet_manager.cc:1367] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20240502 08:18:32.790549 17663 tablet_bootstrap.cc:654] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:32.790918 17651 raft_consensus.cc:348] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:32.791146 17651 raft_consensus.cc:374] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:32.791244 17651 raft_consensus.cc:729] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:32.791476 17651 consensus_queue.cc:260] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:32.792132 17651 ts_tablet_manager.cc:1398] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:32.792663 17651 tablet_bootstrap.cc:492] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:32.794009 17655 tablet_bootstrap.cc:492] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:32.794216 17655 ts_tablet_manager.cc:1367] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.001s
I20240502 08:18:32.795056 17655 raft_consensus.cc:348] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.795328 17655 raft_consensus.cc:374] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:32.796167 17655 raft_consensus.cc:729] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:32.796496 17655 consensus_queue.cc:260] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.797534 17651 tablet_bootstrap.cc:654] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:32.798797 17655 ts_tablet_manager.cc:1398] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:18:32.805490 17654 ts_tablet_manager.cc:1398] T e8e1b3aa05d146048d50ae5c38f6f56b P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 08:18:32.809805 17651 tablet_bootstrap.cc:492] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:32.810096 17651 ts_tablet_manager.cc:1367] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20240502 08:18:32.810978 17651 raft_consensus.cc:348] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.811252 17651 raft_consensus.cc:374] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:32.811404 17651 raft_consensus.cc:729] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:32.811695 17654 tablet_bootstrap.cc:492] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:32.812677 17651 consensus_queue.cc:260] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.816241 17654 tablet_bootstrap.cc:654] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:32.816751 17651 ts_tablet_manager.cc:1398] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:18:32.829288 17663 tablet_bootstrap.cc:492] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:32.829448 17663 ts_tablet_manager.cc:1367] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.042s user 0.020s sys 0.007s
I20240502 08:18:32.830192 17663 raft_consensus.cc:348] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.830518 17663 raft_consensus.cc:374] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:32.830650 17663 raft_consensus.cc:729] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:32.830906 17663 consensus_queue.cc:260] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.832522 17663 ts_tablet_manager.cc:1398] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:32.833246 17654 tablet_bootstrap.cc:492] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: No bootstrap required, opened a new log
I20240502 08:18:32.833400 17654 ts_tablet_manager.cc:1367] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20240502 08:18:32.834112 17654 raft_consensus.cc:348] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.834336 17654 raft_consensus.cc:374] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:32.834508 17654 raft_consensus.cc:729] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:32.834856 17654 consensus_queue.cc:260] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.835448 17654 ts_tablet_manager.cc:1398] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:18:32.921558 17650 raft_consensus.cc:482] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:32.921697 17650 raft_consensus.cc:504] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:32.922284 17650 leader_election.cc:288] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:32.922655 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f66921e73457eb07acec0670b2426" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:32.922693 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f66921e73457eb07acec0670b2426" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:32.922852 12859 raft_consensus.cc:2454] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:32.922868 12990 raft_consensus.cc:2454] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:32.923229 15419 leader_election.cc:302] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:32.923395 17423 raft_consensus.cc:482] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:32.923476 17650 raft_consensus.cc:2781] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:32.923583 17650 raft_consensus.cc:482] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:32.923671 17650 raft_consensus.cc:3037] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.923575 17423 raft_consensus.cc:504] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.924197 17423 leader_election.cc:288] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:32.924611 12728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e1b3aa05d146048d50ae5c38f6f56b" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:32.924600 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e1b3aa05d146048d50ae5c38f6f56b" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:18:32.924798 12728 raft_consensus.cc:2454] T e8e1b3aa05d146048d50ae5c38f6f56b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:32.924829 13121 raft_consensus.cc:2454] T e8e1b3aa05d146048d50ae5c38f6f56b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 0.
I20240502 08:18:32.925221 15819 leader_election.cc:302] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 29b035434087435cbaf4ebcac98eedee; no voters:
I20240502 08:18:32.925642 17457 raft_consensus.cc:2781] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:32.925727 17457 raft_consensus.cc:482] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:32.925798 17457 raft_consensus.cc:3037] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.926203 17650 raft_consensus.cc:504] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:32.926800 17650 leader_election.cc:288] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:32.927204 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f66921e73457eb07acec0670b2426" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:32.927270 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f66921e73457eb07acec0670b2426" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:32.927381 12990 raft_consensus.cc:3037] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.927484 12859 raft_consensus.cc:3037] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.928293 17457 raft_consensus.cc:504] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.928828 17457 leader_election.cc:288] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:32.929595 12728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e1b3aa05d146048d50ae5c38f6f56b" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:32.929805 12728 raft_consensus.cc:3037] T e8e1b3aa05d146048d50ae5c38f6f56b P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.929996 12859 raft_consensus.cc:2454] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:32.931094 13121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e1b3aa05d146048d50ae5c38f6f56b" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:32.931295 13121 raft_consensus.cc:3037] T e8e1b3aa05d146048d50ae5c38f6f56b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.932495 12728 raft_consensus.cc:2454] T e8e1b3aa05d146048d50ae5c38f6f56b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:32.932847 15819 leader_election.cc:302] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 29b035434087435cbaf4ebcac98eedee; no voters:
I20240502 08:18:32.933176 17457 raft_consensus.cc:2781] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:32.933176 17647 raft_consensus.cc:482] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:32.933331 17457 raft_consensus.cc:686] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:18:32.933302 17647 raft_consensus.cc:504] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:32.933563 17457 consensus_queue.cc:237] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.933887 17647 leader_election.cc:288] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:32.933871 13121 raft_consensus.cc:2454] T e8e1b3aa05d146048d50ae5c38f6f56b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:18:32.934552 17227 raft_consensus.cc:482] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:32.934713 17227 raft_consensus.cc:504] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:32.935308 17227 leader_election.cc:288] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:32.935760 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30001d4bd6e943f88d28894ef59cbe18" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:32.935938 15886 raft_consensus.cc:2454] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:32.936332 15420 leader_election.cc:302] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:32.937189 12990 raft_consensus.cc:2454] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:32.937292 15419 leader_election.cc:302] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db7893bb29a047b3b82cc60bfd8930a3, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:32.937316 17227 raft_consensus.cc:2781] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:32.937422 17227 raft_consensus.cc:482] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:32.937521 17227 raft_consensus.cc:3037] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.937747 17577 raft_consensus.cc:2781] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:32.937891 17577 raft_consensus.cc:686] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:32.938184 17577 consensus_queue.cc:237] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:32.939508 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30001d4bd6e943f88d28894ef59cbe18" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:32.939734 12859 raft_consensus.cc:2454] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 0.
I20240502 08:18:32.939978 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30001d4bd6e943f88d28894ef59cbe18" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:32.940214 17227 raft_consensus.cc:504] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:32.940302 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30001d4bd6e943f88d28894ef59cbe18" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:32.940471 15886 raft_consensus.cc:2454] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:32.940846 12797 leader_election.cc:302] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:32.941102 17647 raft_consensus.cc:2781] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:32.941192 17647 raft_consensus.cc:482] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:32.941273 17647 raft_consensus.cc:3037] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.942893 17227 leader_election.cc:288] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:32.943079 15488 raft_consensus.cc:2379] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e1dd14f7ca764a1490df419f13f2804a in current term 1: Already voted for candidate db7893bb29a047b3b82cc60bfd8930a3 in this term.
I20240502 08:18:32.943159 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30001d4bd6e943f88d28894ef59cbe18" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:32.943317 15886 raft_consensus.cc:3037] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.943636 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30001d4bd6e943f88d28894ef59cbe18" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:32.943629 17647 raft_consensus.cc:504] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:32.945199 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30001d4bd6e943f88d28894ef59cbe18" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:32.945962 15886 raft_consensus.cc:2454] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
I20240502 08:18:32.946477 15420 leader_election.cc:302] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:32.946729 17227 raft_consensus.cc:2781] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:32.946892 17227 raft_consensus.cc:686] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:32.946923 17647 leader_election.cc:288] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:32.947125 17227 consensus_queue.cc:237] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:32.947873 12859 raft_consensus.cc:2379] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate db7893bb29a047b3b82cc60bfd8930a3 in current term 1: Already voted for candidate e1dd14f7ca764a1490df419f13f2804a in this term.
I20240502 08:18:32.947810 12584 catalog_manager.cc:5484] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:32.948473 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30001d4bd6e943f88d28894ef59cbe18" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:32.948920 12796 leader_election.cc:302] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1dd14f7ca764a1490df419f13f2804a; no voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:32.950628 17647 raft_consensus.cc:2726] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 08:18:32.960968 17520 raft_consensus.cc:482] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:32.961104 17520 raft_consensus.cc:504] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.961174 12588 catalog_manager.cc:5484] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:32.962347 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6afd792544a45acb79669ed249559a9" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:32.962577 12859 raft_consensus.cc:2454] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:32.962378 12728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6afd792544a45acb79669ed249559a9" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:18:32.962916 12728 raft_consensus.cc:2454] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:32.962919 17520 leader_election.cc:288] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:32.963378 15688 leader_election.cc:302] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:32.963658 17520 raft_consensus.cc:2781] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:32.963821 17520 raft_consensus.cc:482] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:32.963939 17520 raft_consensus.cc:3037] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.967974 12588 catalog_manager.cc:5484] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:32.968181 17520 raft_consensus.cc:504] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.968884 17520 leader_election.cc:288] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:32.969434 12859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6afd792544a45acb79669ed249559a9" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:32.969651 12859 raft_consensus.cc:3037] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.971529 12728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6afd792544a45acb79669ed249559a9" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:18:32.971745 12728 raft_consensus.cc:3037] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:32.972232 12859 raft_consensus.cc:2454] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:32.974190 12728 raft_consensus.cc:2454] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:32.974517 15688 leader_election.cc:302] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:32.975181 17520 raft_consensus.cc:2781] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:32.975349 17520 raft_consensus.cc:686] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:32.975596 17520 consensus_queue.cc:237] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:32.979424 12588 catalog_manager.cc:5484] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:33.128073 12588 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 08:18:33.148474 12861 raft_consensus.cc:1261] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:33.149055 17275 consensus_queue.cc:1035] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:33.151624 12728 raft_consensus.cc:1261] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:33.152412 17275 consensus_queue.cc:1035] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:33.156715 17679 tablet.cc:1722] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:33.157330 17679 tablet.cc:1620] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:33.158676 12584 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet a6afd792544a45acb79669ed249559a9 (table rebalancer_dynamic_table [id=2c10382fe86b483cab4494831f51bb62]). Expected version 1 got 0
I20240502 08:18:33.160286 17678 tablet.cc:1722] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:33.161223 12588 catalog_manager.cc:5576] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933) does not have the latest schema for tablet a6afd792544a45acb79669ed249559a9 (table rebalancer_dynamic_table [id=2c10382fe86b483cab4494831f51bb62]). Expected version 1 got 0
I20240502 08:18:33.162813 17678 tablet.cc:1620] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:33.170993 17680 tablet.cc:1722] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 08:18:33.171520 12583 catalog_manager.cc:5576] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183) does not have the latest schema for tablet a6afd792544a45acb79669ed249559a9 (table rebalancer_dynamic_table [id=2c10382fe86b483cab4494831f51bb62]). Expected version 1 got 0
I20240502 08:18:33.171710 17680 tablet.cc:1620] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:33.172905 12588 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=2c10382fe86b483cab4494831f51bb62] alter complete (version 1)
I20240502 08:18:33.277987 12588 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 08:18:33.287575 17679 tablet.cc:1722] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:33.288079 17679 tablet.cc:1620] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: MemRowSet was empty: no flush needed.
I20240502 08:18:33.289724 17678 tablet.cc:1722] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:33.290153 17678 tablet.cc:1620] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: MemRowSet was empty: no flush needed.
I20240502 08:18:33.300305 12588 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table [id=2c10382fe86b483cab4494831f51bb62] alter complete (version 2)
I20240502 08:18:33.303395 17680 tablet.cc:1722] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 08:18:33.303803 17680 tablet.cc:1620] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: MemRowSet was empty: no flush needed.
I20240502 08:18:33.331111 17577 consensus_queue.cc:1035] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:33.352129 17227 consensus_queue.cc:1035] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:33.370765 17227 consensus_queue.cc:1035] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:33.375391 17650 consensus_queue.cc:1035] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:33.403992 12583 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 08:18:33.411324 17678 tablet.cc:1722] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:33.412060 17680 tablet.cc:1722] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:33.416656 17679 tablet.cc:1722] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 08:18:33.422669 12583 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: rebalancer_dynamic_table_new_name [id=2c10382fe86b483cab4494831f51bb62] alter complete (version 3)
I20240502 08:18:33.451154 17423 consensus_queue.cc:1035] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:33.458371 17457 consensus_queue.cc:1035] T e8e1b3aa05d146048d50ae5c38f6f56b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:33.489248 12583 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:18:33.498594 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 815d79a2bbba49ea9e135d94c57a3390 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=6aba7496ff9b4c90ae1f70fee8b918e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:33.499056 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815d79a2bbba49ea9e135d94c57a3390. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.499182 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 815d79a2bbba49ea9e135d94c57a3390 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=6aba7496ff9b4c90ae1f70fee8b918e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:33.499570 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815d79a2bbba49ea9e135d94c57a3390. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.499922 12841 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3cc08d9534400ab7ee8db08e6e7ee0 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=6aba7496ff9b4c90ae1f70fee8b918e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:33.499876 12842 tablet_service.cc:1467] Processing CreateTablet for tablet 815d79a2bbba49ea9e135d94c57a3390 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=6aba7496ff9b4c90ae1f70fee8b918e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:33.500315 12841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3cc08d9534400ab7ee8db08e6e7ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.500548 12842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815d79a2bbba49ea9e135d94c57a3390. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.501224 12970 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3cc08d9534400ab7ee8db08e6e7ee0 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=6aba7496ff9b4c90ae1f70fee8b918e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:33.501578 12970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3cc08d9534400ab7ee8db08e6e7ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.505412 15601 tablet_service.cc:1467] Processing CreateTablet for tablet 2e3cc08d9534400ab7ee8db08e6e7ee0 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=6aba7496ff9b4c90ae1f70fee8b918e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:33.505792 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3cc08d9534400ab7ee8db08e6e7ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.505733 15600 tablet_service.cc:1467] Processing CreateTablet for tablet 188e9a523fee46a08a4c6180058ed923 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=6aba7496ff9b4c90ae1f70fee8b918e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:33.506034 15600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e9a523fee46a08a4c6180058ed923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.515408 12972 tablet_service.cc:1467] Processing CreateTablet for tablet 188e9a523fee46a08a4c6180058ed923 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=6aba7496ff9b4c90ae1f70fee8b918e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:33.515815 12972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e9a523fee46a08a4c6180058ed923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.516201 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 188e9a523fee46a08a4c6180058ed923 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=6aba7496ff9b4c90ae1f70fee8b918e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:33.516579 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e9a523fee46a08a4c6180058ed923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.521699 17695 tablet_bootstrap.cc:492] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:33.524138 17695 tablet_bootstrap.cc:654] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:33.527103 17696 tablet_bootstrap.cc:492] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:33.527310 17693 tablet_bootstrap.cc:492] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:33.528873 17696 tablet_bootstrap.cc:654] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:33.530097 17694 tablet_bootstrap.cc:492] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:33.530241 12583 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:33.530447 17693 tablet_bootstrap.cc:654] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:33.530486 12583 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39466:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 08:18:33.532477 17694 tablet_bootstrap.cc:654] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:33.533500 17696 tablet_bootstrap.cc:492] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:33.533695 17696 ts_tablet_manager.cc:1367] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240502 08:18:33.533923 12583 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 17be716fc99c4d588d7726089a2116d2: Sending DeleteTablet for 3 replicas of tablet a6afd792544a45acb79669ed249559a9
I20240502 08:18:33.532934 17693 tablet_bootstrap.cc:492] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:33.534734 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet a6afd792544a45acb79669ed249559a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:33 UTC) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:33.534765 17693 ts_tablet_manager.cc:1367] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240502 08:18:33.535032 12708 tablet_service.cc:1514] Processing DeleteTablet for tablet a6afd792544a45acb79669ed249559a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:33 UTC) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:33.535557 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet a6afd792544a45acb79669ed249559a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 08:18:33 UTC) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:33.537700 17703 tablet_replica.cc:330] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:33.537662 17693 raft_consensus.cc:348] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.538350 17695 tablet_bootstrap.cc:492] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:33.538559 17695 ts_tablet_manager.cc:1367] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 08:18:33.538573 17693 raft_consensus.cc:374] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:33.538661 17693 raft_consensus.cc:729] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:33.537945 17703 raft_consensus.cc:2229] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Raft consensus shutting down.
I20240502 08:18:33.539105 17703 raft_consensus.cc:2258] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:33.539427 17693 consensus_queue.cc:260] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.540012 17693 ts_tablet_manager.cc:1398] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 08:18:33.541075 17693 tablet_bootstrap.cc:492] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:33.542543 17693 tablet_bootstrap.cc:654] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:33.546841 17696 raft_consensus.cc:348] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:33.547363 17696 raft_consensus.cc:374] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:33.547585 17696 raft_consensus.cc:729] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:33.548012 17696 consensus_queue.cc:260] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:33.548564 17695 raft_consensus.cc:348] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.548787 17695 raft_consensus.cc:374] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:33.548846 17695 raft_consensus.cc:729] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:33.549072 17695 consensus_queue.cc:260] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.549273 17696 ts_tablet_manager.cc:1398] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 08:18:33.549852 17695 ts_tablet_manager.cc:1398] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 08:18:33.550175 17695 tablet_bootstrap.cc:492] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:33.556540 17703 ts_tablet_manager.cc:1875] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:33.556571 17702 tablet_replica.cc:330] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:33.556784 17702 raft_consensus.cc:2229] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:33.557122 17702 raft_consensus.cc:2258] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:33.558029 17702 ts_tablet_manager.cc:1875] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:33.561455 17696 tablet_bootstrap.cc:492] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:33.562294 17695 tablet_bootstrap.cc:654] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:33.564785 17696 tablet_bootstrap.cc:654] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:33.566699 17694 tablet_bootstrap.cc:492] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:33.566921 17694 ts_tablet_manager.cc:1367] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.007s
I20240502 08:18:33.567164 17701 tablet_replica.cc:330] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:33.567595 17701 raft_consensus.cc:2229] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:33.567862 17701 raft_consensus.cc:2258] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:33.567762 17694 raft_consensus.cc:348] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.569116 17694 raft_consensus.cc:374] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:33.569275 17694 raft_consensus.cc:729] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:33.569545 17694 consensus_queue.cc:260] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.572678 17694 ts_tablet_manager.cc:1398] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:18:33.573140 17694 tablet_bootstrap.cc:492] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:33.573542 17703 ts_tablet_manager.cc:1888] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:33.577080 17695 tablet_bootstrap.cc:492] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4: No bootstrap required, opened a new log
I20240502 08:18:33.577302 17695 ts_tablet_manager.cc:1367] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.000s
I20240502 08:18:33.578336 17696 tablet_bootstrap.cc:492] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:33.578260 17695 raft_consensus.cc:348] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:33.578549 17696 ts_tablet_manager.cc:1367] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240502 08:18:33.578606 17695 raft_consensus.cc:374] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:33.578744 17695 raft_consensus.cc:729] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:33.579118 17398 raft_consensus.cc:482] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:33.579073 17695 consensus_queue.cc:260] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:33.579270 17398 raft_consensus.cc:504] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:33.579926 17695 ts_tablet_manager.cc:1398] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:33.579959 17398 leader_election.cc:288] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:33.582001 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e9a523fee46a08a4c6180058ed923" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:33.582162 17694 tablet_bootstrap.cc:654] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:33.582245 12992 raft_consensus.cc:2454] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 0.
I20240502 08:18:33.582721 15554 leader_election.cc:302] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:18:33.583266 17263 raft_consensus.cc:2781] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:33.583348 17263 raft_consensus.cc:482] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:33.583436 17263 raft_consensus.cc:3037] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:33.584553 17702 ts_tablet_manager.cc:1888] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:33.584699 17702 log.cc:1196] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/a6afd792544a45acb79669ed249559a9
I20240502 08:18:33.585060 17702 ts_tablet_manager.cc:1909] T a6afd792544a45acb79669ed249559a9 P 29b035434087435cbaf4ebcac98eedee: Deleting consensus metadata
I20240502 08:18:33.568972 17701 ts_tablet_manager.cc:1875] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 08:18:33.585870 17263 raft_consensus.cc:504] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:33.586870 17263 leader_election.cc:288] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:33.588925 17696 raft_consensus.cc:348] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.589293 17696 raft_consensus.cc:374] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:33.589830 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet a6afd792544a45acb79669ed249559a9 (table rebalancer_dynamic_table_new_name [id=2c10382fe86b483cab4494831f51bb62]) successfully deleted
I20240502 08:18:33.590210 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e9a523fee46a08a4c6180058ed923" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:33.590394 12992 raft_consensus.cc:3037] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:33.591029 17696 raft_consensus.cc:729] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:33.591367 17696 consensus_queue.cc:260] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.592417 17701 ts_tablet_manager.cc:1888] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 08:18:33.592527 17701 log.cc:1196] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/a6afd792544a45acb79669ed249559a9
I20240502 08:18:33.592942 17701 ts_tablet_manager.cc:1909] T a6afd792544a45acb79669ed249559a9 P e1dd14f7ca764a1490df419f13f2804a: Deleting consensus metadata
I20240502 08:18:33.593179 12992 raft_consensus.cc:2454] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in term 1.
I20240502 08:18:33.581518 17703 log.cc:1196] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/a6afd792544a45acb79669ed249559a9
I20240502 08:18:33.582324 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e9a523fee46a08a4c6180058ed923" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:18:33.588552 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e9a523fee46a08a4c6180058ed923" candidate_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:33.594625 17694 tablet_bootstrap.cc:492] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a: No bootstrap required, opened a new log
I20240502 08:18:33.594827 17694 ts_tablet_manager.cc:1367] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.003s
W20240502 08:18:33.595693 15553 leader_election.cc:341] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:33.595744 17694 raft_consensus.cc:348] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
W20240502 08:18:33.595919 15553 leader_election.cc:341] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:33.596071 17694 raft_consensus.cc:374] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:33.596180 17694 raft_consensus.cc:729] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:33.596477 15554 leader_election.cc:302] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters: 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:33.596698 17694 consensus_queue.cc:260] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.596868 17703 ts_tablet_manager.cc:1909] T a6afd792544a45acb79669ed249559a9 P 9333cb10227b4801b4fa7c24e22e131b: Deleting consensus metadata
I20240502 08:18:33.596863 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet a6afd792544a45acb79669ed249559a9 (table rebalancer_dynamic_table_new_name [id=2c10382fe86b483cab4494831f51bb62]) successfully deleted
I20240502 08:18:33.597632 17694 ts_tablet_manager.cc:1398] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:33.597728 17398 raft_consensus.cc:2781] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:33.604177 17696 ts_tablet_manager.cc:1398] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240502 08:18:33.604895 17696 tablet_bootstrap.cc:492] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:33.607957 17696 tablet_bootstrap.cc:654] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:33.609226 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet a6afd792544a45acb79669ed249559a9 (table rebalancer_dynamic_table_new_name [id=2c10382fe86b483cab4494831f51bb62]) successfully deleted
I20240502 08:18:33.611568 17693 tablet_bootstrap.cc:492] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b: No bootstrap required, opened a new log
I20240502 08:18:33.611730 17693 ts_tablet_manager.cc:1367] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.071s user 0.027s sys 0.022s
I20240502 08:18:33.612326 17693 raft_consensus.cc:348] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:33.612512 17693 raft_consensus.cc:374] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:33.612588 17693 raft_consensus.cc:729] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:33.612826 17693 consensus_queue.cc:260] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:33.613325 17693 ts_tablet_manager.cc:1398] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:33.615630 17398 raft_consensus.cc:686] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Leader. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:18:33.615967 17398 consensus_queue.cc:237] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:33.620054 17696 tablet_bootstrap.cc:492] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: No bootstrap required, opened a new log
I20240502 08:18:33.620335 17696 ts_tablet_manager.cc:1367] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 08:18:33.621160 17696 raft_consensus.cc:348] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.621541 17696 raft_consensus.cc:374] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:33.621690 17696 raft_consensus.cc:729] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:33.622031 17696 consensus_queue.cc:260] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.622733 17696 ts_tablet_manager.cc:1398] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:33.627441 12583 catalog_manager.cc:5484] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6). New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:33.636873 12583 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39466:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 08:18:33.641346 17682 raft_consensus.cc:482] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:33.641523 17682 raft_consensus.cc:504] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.642385 17682 leader_election.cc:288] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:33.642997 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3cc08d9534400ab7ee8db08e6e7ee0" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:33.643227 15621 raft_consensus.cc:2454] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:33.643829 12798 leader_election.cc:302] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:33.644685 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3cc08d9534400ab7ee8db08e6e7ee0" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:33.644896 12992 raft_consensus.cc:2454] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 0.
I20240502 08:18:33.646687 15865 tablet_service.cc:1467] Processing CreateTablet for tablet 51fe8c8f38d4492194f6e3c83f25a60b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=89c263bab2dc46069eb3fa03deb23175]), partition=
I20240502 08:18:33.646696 13102 tablet_service.cc:1467] Processing CreateTablet for tablet 51fe8c8f38d4492194f6e3c83f25a60b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=89c263bab2dc46069eb3fa03deb23175]), partition=
I20240502 08:18:33.646773 15468 tablet_service.cc:1467] Processing CreateTablet for tablet 51fe8c8f38d4492194f6e3c83f25a60b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=89c263bab2dc46069eb3fa03deb23175]), partition=
I20240502 08:18:33.646848 17682 raft_consensus.cc:2781] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:33.646957 17682 raft_consensus.cc:482] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:33.647055 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fe8c8f38d4492194f6e3c83f25a60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.647073 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fe8c8f38d4492194f6e3c83f25a60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.647068 17682 raft_consensus.cc:3037] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:33.647121 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fe8c8f38d4492194f6e3c83f25a60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.650321 17275 raft_consensus.cc:482] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:33.650558 17275 raft_consensus.cc:504] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.651342 17275 leader_election.cc:288] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:33.657166 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d79a2bbba49ea9e135d94c57a3390" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:33.657423 15621 raft_consensus.cc:2454] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:33.657897 15690 leader_election.cc:302] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:18:33.658496 17520 raft_consensus.cc:2781] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:33.658617 17520 raft_consensus.cc:482] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:33.658725 17520 raft_consensus.cc:3037] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:33.661337 17520 raft_consensus.cc:504] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.666781 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d79a2bbba49ea9e135d94c57a3390" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:18:33.667021 12861 raft_consensus.cc:2454] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 0.
I20240502 08:18:33.667857 17520 leader_election.cc:288] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:33.669481 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d79a2bbba49ea9e135d94c57a3390" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:33.669777 12861 raft_consensus.cc:3037] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:33.672735 12861 raft_consensus.cc:2454] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:33.675638 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815d79a2bbba49ea9e135d94c57a3390" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:33.675875 15621 raft_consensus.cc:3037] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:33.679277 15621 raft_consensus.cc:2454] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9333cb10227b4801b4fa7c24e22e131b in term 1.
I20240502 08:18:33.679750 15688 leader_election.cc:302] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:33.680323 17681 raft_consensus.cc:2781] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:33.680460 17681 raft_consensus.cc:686] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Becoming Leader. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Running, Role: LEADER
I20240502 08:18:33.680802 17681 consensus_queue.cc:237] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.649837 17682 raft_consensus.cc:504] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.692632 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3cc08d9534400ab7ee8db08e6e7ee0" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:33.692862 12992 raft_consensus.cc:3037] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:33.696077 12992 raft_consensus.cc:2454] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:33.699401 12796 leader_election.cc:302] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:33.699746 17647 raft_consensus.cc:2781] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:33.702801 17682 leader_election.cc:288] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 1 election: Requested vote from peers 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:33.704566 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e3cc08d9534400ab7ee8db08e6e7ee0" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:18:33.704823 15622 raft_consensus.cc:3037] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:33.707801 15622 raft_consensus.cc:2454] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:18:33.721669 12583 catalog_manager.cc:5484] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7). New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:33.723412 17647 raft_consensus.cc:686] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:18:33.723829 17647 consensus_queue.cc:237] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:33.748824 17713 tablet_bootstrap.cc:492] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:33.757097 17713 tablet_bootstrap.cc:654] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:33.770759 17712 tablet_bootstrap.cc:492] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:33.771606 17711 tablet_bootstrap.cc:492] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:33.778182 17712 tablet_bootstrap.cc:654] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:33.778753 17711 tablet_bootstrap.cc:654] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:33.779742 17713 tablet_bootstrap.cc:492] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f: No bootstrap required, opened a new log
I20240502 08:18:33.780882 17712 tablet_bootstrap.cc:492] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3: No bootstrap required, opened a new log
I20240502 08:18:33.781369 17713 ts_tablet_manager.cc:1367] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240502 08:18:33.782913 17712 ts_tablet_manager.cc:1367] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 08:18:33.782977 12584 catalog_manager.cc:5484] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:33.784577 17713 raft_consensus.cc:348] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:33.785220 17712 raft_consensus.cc:348] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:33.785498 17713 raft_consensus.cc:374] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:33.785643 17713 raft_consensus.cc:729] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:33.787536 17712 raft_consensus.cc:374] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:33.787636 17712 raft_consensus.cc:729] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:33.787853 17712 consensus_queue.cc:260] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:33.788386 17712 ts_tablet_manager.cc:1398] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 08:18:33.788594 17713 consensus_queue.cc:260] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:33.789211 17713 ts_tablet_manager.cc:1398] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:18:33.792769 17711 tablet_bootstrap.cc:492] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98: No bootstrap required, opened a new log
I20240502 08:18:33.792944 17711 ts_tablet_manager.cc:1367] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240502 08:18:33.794180 17711 raft_consensus.cc:348] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:33.794428 17711 raft_consensus.cc:374] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:33.794591 17711 raft_consensus.cc:729] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:33.796284 17711 consensus_queue.cc:260] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:33.798454 17711 ts_tablet_manager.cc:1398] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:18:34.002610 17549 raft_consensus.cc:482] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:34.002758 17549 raft_consensus.cc:504] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:34.003269 17549 leader_election.cc:288] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:34.003609 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fe8c8f38d4492194f6e3c83f25a60b" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:18:34.003669 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fe8c8f38d4492194f6e3c83f25a60b" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:18:34.003861 15488 raft_consensus.cc:2454] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:34.003865 15887 raft_consensus.cc:2454] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 0.
I20240502 08:18:34.004267 13057 leader_election.cc:302] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
I20240502 08:18:34.004530 17625 raft_consensus.cc:2781] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:34.004630 17625 raft_consensus.cc:482] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:34.004731 17625 raft_consensus.cc:3037] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:34.006999 17625 raft_consensus.cc:504] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:34.007511 17625 leader_election.cc:288] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:34.007903 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fe8c8f38d4492194f6e3c83f25a60b" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:34.008105 15488 raft_consensus.cc:3037] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:34.009899 15488 raft_consensus.cc:2454] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:34.010284 13056 leader_election.cc:302] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:34.010552 17625 raft_consensus.cc:2781] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:34.010663 17625 raft_consensus.cc:686] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:18:34.010865 17625 consensus_queue.cc:237] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:34.014302 15886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fe8c8f38d4492194f6e3c83f25a60b" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:18:34.014528 15886 raft_consensus.cc:3037] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:34.015329 12588 catalog_manager.cc:5484] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:34.016465 15886 raft_consensus.cc:2454] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:18:34.034638 17520 raft_consensus.cc:482] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:34.034780 17520 raft_consensus.cc:504] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:34.035317 17520 leader_election.cc:288] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:34.035681 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e9a523fee46a08a4c6180058ed923" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" is_pre_election: true
I20240502 08:18:34.036073 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e9a523fee46a08a4c6180058ed923" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:18:34.036314 12992 raft_consensus.cc:2379] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9333cb10227b4801b4fa7c24e22e131b in current term 1: Already voted for candidate 8d45ab45b8eb4d03a4bb9cb88b78a98b in this term.
I20240502 08:18:34.036824 15688 leader_election.cc:302] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b; no voters: 60b6e265e7754b7187307d05ef405dd4, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:34.037101 17520 raft_consensus.cc:3037] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:34.039875 17520 raft_consensus.cc:2726] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:34.084497 17520 consensus_queue.cc:1035] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:34.096988 17520 consensus_queue.cc:1035] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:34.115746 17708 consensus_queue.cc:1035] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:34.122052 17708 consensus_queue.cc:1035] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:34.139544 17647 consensus_queue.cc:1035] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:34.144958 17715 consensus_queue.cc:1035] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:34.410842 17549 consensus_queue.cc:1035] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:34.428058 17739 consensus_queue.cc:1035] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:34.885867 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:34.886322 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:34.899811 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:34.910027 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:34.927129 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:34.993623 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:35.002573 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:35.040156 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:35.535024 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:35.541572 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:35.561908 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:35.584172 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:35.600616 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:35.600893 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:35.635804 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:35.646821 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:36.141176 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:36.148234 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:36.174844 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:36.220139 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:36.220615 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:36.239790 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:36.247790 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:36.269547 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:18:36.361160 15818 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.7:38851, user_credentials={real_user=slave}} blocked reactor thread for 64964.5us
I20240502 08:18:36.635301 12728 consensus_queue.cc:237] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
W20240502 08:18:36.637984 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c37ff40706043e9adc4da4a6da1947a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:36.639017 12861 raft_consensus.cc:1261] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:36.639582 17154 consensus_queue.cc:1035] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:36.639873 13121 raft_consensus.cc:1261] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:36.640362 17154 consensus_queue.cc:1035] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:36.644207 17619 raft_consensus.cc:2932] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:36.645823 13119 raft_consensus.cc:2932] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:36.648429 12861 raft_consensus.cc:2932] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:36.650614 12587 catalog_manager.cc:5484] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:36.698241 15755 consensus_queue.cc:237] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:36.701171 15886 raft_consensus.cc:1261] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:36.701710 15688 consensus_peers.cc:485] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Couldn't send request to peer 29b035434087435cbaf4ebcac98eedee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 106ebc4b0ddd4acc824ae37042907a89. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:36.701740 15488 raft_consensus.cc:1261] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:36.701903 17275 consensus_queue.cc:1035] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:36.702268 17946 consensus_queue.cc:1035] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:36.705128 17275 raft_consensus.cc:2932] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:36.705708 15488 raft_consensus.cc:2932] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:36.710147 15886 raft_consensus.cc:2932] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:36.716228 12587 catalog_manager.cc:5484] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:36.775444 15488 consensus_queue.cc:237] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:36.778342 12861 raft_consensus.cc:1261] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:36.778841 15419 consensus_peers.cc:485] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): Couldn't send request to peer 60b6e265e7754b7187307d05ef405dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5cd9e39bfc740b78492d369532d0c66. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:36.779140 17735 consensus_queue.cc:1035] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:36.779399 15754 raft_consensus.cc:1261] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:36.780081 18009 consensus_queue.cc:1035] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:36.782480 17735 raft_consensus.cc:2932] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:36.785873 15754 raft_consensus.cc:2932] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:36.788861 12584 catalog_manager.cc:5484] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:36.790078 12861 raft_consensus.cc:2932] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:36.841068 15622 consensus_queue.cc:237] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:36.843761 15488 raft_consensus.cc:1261] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:36.844259 17398 consensus_queue.cc:1035] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:36.844616 15553 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b -> Peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Couldn't send request to peer 9333cb10227b4801b4fa7c24e22e131b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9081ea0b98443d38f42c14cd626f0d0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:36.851289 12859 raft_consensus.cc:1261] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:36.851737 18029 consensus_queue.cc:1035] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:36.855058 17729 raft_consensus.cc:2932] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:36.855609 12859 raft_consensus.cc:2932] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:36.862771 12588 catalog_manager.cc:5484] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:36.866135 15488 raft_consensus.cc:2932] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:36.912838 12728 consensus_queue.cc:237] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:36.914959 12859 raft_consensus.cc:1261] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:36.915196 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab808742ce8746ae8e9b8fbff435e7be. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:36.915272 13119 raft_consensus.cc:1261] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:36.915458 17154 consensus_queue.cc:1035] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:36.915710 17619 consensus_queue.cc:1035] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:36.918366 17154 raft_consensus.cc:2932] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:36.919024 13119 raft_consensus.cc:2932] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:36.926275 12584 catalog_manager.cc:5484] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:36.927933 12859 raft_consensus.cc:2932] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:36.962755 13119 consensus_queue.cc:237] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:36.965538 15754 raft_consensus.cc:1261] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:36.965540 15886 raft_consensus.cc:1261] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:36.966094 18067 consensus_queue.cc:1035] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:36.966591 17549 consensus_queue.cc:1035] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:36.966969 13056 consensus_peers.cc:485] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Couldn't send request to peer 29b035434087435cbaf4ebcac98eedee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dcaab7ded074ca38dda02c3bdf4c5e9. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:36.970022 17549 raft_consensus.cc:2932] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:36.970578 15886 raft_consensus.cc:2932] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:36.974519 15754 raft_consensus.cc:2932] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:36.976025 12587 catalog_manager.cc:5484] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:37.013031 13119 consensus_queue.cc:237] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:37.015928 12990 raft_consensus.cc:1261] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:37.016175 13056 consensus_peers.cc:485] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Couldn't send request to peer db7893bb29a047b3b82cc60bfd8930a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf504f2522a147c198f99debdfe03d1e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.016559 17549 consensus_queue.cc:1035] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:37.016719 12859 raft_consensus.cc:1261] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:37.017381 17549 consensus_queue.cc:1035] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:37.027441 17549 raft_consensus.cc:2932] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:37.028239 12990 raft_consensus.cc:2932] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:37.035863 12859 raft_consensus.cc:2932] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:37.036026 12588 catalog_manager.cc:5484] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:37.084168 15754 consensus_queue.cc:237] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:37.086321 15621 raft_consensus.cc:1261] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:37.086522 15688 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): Couldn't send request to peer 60b6e265e7754b7187307d05ef405dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea85ef42ac714074a10f2a5ece4ddf8e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.086596 13119 raft_consensus.cc:1261] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:37.087055 17520 consensus_queue.cc:1035] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:37.087365 17275 consensus_queue.cc:1035] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:37.090171 17520 raft_consensus.cc:2932] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:37.090914 15621 raft_consensus.cc:2932] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:37.094110 13119 raft_consensus.cc:2932] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:37.097572 12584 catalog_manager.cc:5484] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:37.141857 15488 consensus_queue.cc:237] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:37.143992 12990 raft_consensus.cc:1261] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:37.144026 15754 raft_consensus.cc:1261] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:37.144392 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a55a182e1f54db698ba8060eea80da0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.144537 17735 consensus_queue.cc:1035] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:37.144789 17577 consensus_queue.cc:1035] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:37.151677 17577 raft_consensus.cc:2932] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:37.152230 15754 raft_consensus.cc:2932] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:37.155428 12990 raft_consensus.cc:2932] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:37.157721 12588 catalog_manager.cc:5484] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:37.203449 13119 consensus_queue.cc:237] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
W20240502 08:18:37.210404 13057 consensus_peers.cc:485] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Couldn't send request to peer 10f0e36203914b85b77e78609ffced98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f031f5fedb904eaf83a4902092280295. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.210498 12730 raft_consensus.cc:1261] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:37.210561 12861 raft_consensus.cc:1261] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:37.211148 18067 consensus_queue.cc:1035] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:37.211714 18067 consensus_queue.cc:1035] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:37.221518 18141 ts_tablet_manager.cc:902] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:37.222110 18067 raft_consensus.cc:2932] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:37.222549 12859 raft_consensus.cc:2932] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:37.230943 12727 raft_consensus.cc:2932] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:37.236888 12584 catalog_manager.cc:5484] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:37.255681 18145 ts_tablet_manager.cc:902] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:37.268782 18145 tablet_copy_client.cc:308] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:18:37.269888 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet 3c37ff40706043e9adc4da4a6da1947a from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:55577)
I20240502 08:18:37.270128 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet 3c37ff40706043e9adc4da4a6da1947a from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:55577: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-3c37ff40706043e9adc4da4a6da1947a
I20240502 08:18:37.273447 12751 tablet_copy_source_session.cc:205] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:37.273500 18141 tablet_copy_client.cc:308] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:18:37.275559 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet b5cd9e39bfc740b78492d369532d0c66 from peer 60b6e265e7754b7187307d05ef405dd4 ({username='slave'} at 127.6.5.3:43031)
I20240502 08:18:37.275826 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet b5cd9e39bfc740b78492d369532d0c66 from peer 60b6e265e7754b7187307d05ef405dd4 at {username='slave'} at 127.6.5.3:43031: session id = 60b6e265e7754b7187307d05ef405dd4-b5cd9e39bfc740b78492d369532d0c66
I20240502 08:18:37.279284 15508 tablet_copy_source_session.cc:205] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:37.281716 18141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cd9e39bfc740b78492d369532d0c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:37.284056 18145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c37ff40706043e9adc4da4a6da1947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:37.287758 18141 tablet_copy_client.cc:783] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:37.288034 18141 tablet_copy_client.cc:647] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:37.292886 18166 ts_tablet_manager.cc:902] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee: Initiating tablet copy from peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:37.296372 18141 tablet_copy_client.cc:523] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:37.300244 18145 tablet_copy_client.cc:783] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:37.300607 18145 tablet_copy_client.cc:647] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:37.301034 18141 tablet_bootstrap.cc:492] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:37.303705 18145 tablet_copy_client.cc:523] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:37.308691 18145 tablet_bootstrap.cc:492] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:37.335520 18141 tablet_bootstrap.cc:492] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.337083 18141 tablet_bootstrap.cc:492] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap complete.
I20240502 08:18:37.337348 18141 ts_tablet_manager.cc:1367] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.000s
I20240502 08:18:37.338109 18141 raft_consensus.cc:348] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:37.338335 18141 raft_consensus.cc:729] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:37.339685 18171 ts_tablet_manager.cc:902] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b: Initiating tablet copy from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:37.355458 18173 ts_tablet_manager.cc:902] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:37.356923 18141 consensus_queue.cc:260] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:37.358137 18166 tablet_copy_client.cc:308] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.7:38851
I20240502 08:18:37.359972 15775 tablet_copy_service.cc:140] P 9333cb10227b4801b4fa7c24e22e131b: Received BeginTabletCopySession request for tablet 106ebc4b0ddd4acc824ae37042907a89 from peer 29b035434087435cbaf4ebcac98eedee ({username='slave'} at 127.6.5.1:37709)
I20240502 08:18:37.360196 15775 tablet_copy_service.cc:161] P 9333cb10227b4801b4fa7c24e22e131b: Beginning new tablet copy session on tablet 106ebc4b0ddd4acc824ae37042907a89 from peer 29b035434087435cbaf4ebcac98eedee at {username='slave'} at 127.6.5.1:37709: session id = 29b035434087435cbaf4ebcac98eedee-106ebc4b0ddd4acc824ae37042907a89
I20240502 08:18:37.360417 18141 ts_tablet_manager.cc:1398] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240502 08:18:37.361207 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session 60b6e265e7754b7187307d05ef405dd4-b5cd9e39bfc740b78492d369532d0c66 received from {username='slave'} at 127.6.5.3:43031
I20240502 08:18:37.361358 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session 60b6e265e7754b7187307d05ef405dd4-b5cd9e39bfc740b78492d369532d0c66 on tablet b5cd9e39bfc740b78492d369532d0c66 with peer 60b6e265e7754b7187307d05ef405dd4
I20240502 08:18:37.367677 15775 tablet_copy_source_session.cc:205] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:37.369659 18166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106ebc4b0ddd4acc824ae37042907a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:37.371930 15621 consensus_queue.cc:237] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:37.376567 18166 tablet_copy_client.cc:783] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:37.376838 18166 tablet_copy_client.cc:647] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:37.378892 18166 tablet_copy_client.cc:523] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:37.382076 18166 tablet_bootstrap.cc:492] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
W20240502 08:18:37.383957 15554 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8774c037b901461684d96460e663d17e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.385663 15887 raft_consensus.cc:1261] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:37.386215 17729 consensus_queue.cc:1035] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:37.394217 15488 raft_consensus.cc:1261] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:37.397207 17729 consensus_queue.cc:1035] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:37.397540 18166 tablet_bootstrap.cc:492] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.397918 18166 tablet_bootstrap.cc:492] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee: Bootstrap complete.
I20240502 08:18:37.398152 18166 ts_tablet_manager.cc:1367] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20240502 08:18:37.398877 18166 raft_consensus.cc:348] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:37.399087 18166 raft_consensus.cc:729] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:37.399344 18166 consensus_queue.cc:260] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:37.405472 18166 ts_tablet_manager.cc:1398] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:18:37.410401 15775 tablet_copy_service.cc:342] P 9333cb10227b4801b4fa7c24e22e131b: Request end of tablet copy session 29b035434087435cbaf4ebcac98eedee-106ebc4b0ddd4acc824ae37042907a89 received from {username='slave'} at 127.6.5.1:37709
I20240502 08:18:37.410571 15775 tablet_copy_service.cc:433] P 9333cb10227b4801b4fa7c24e22e131b: ending tablet copy session 29b035434087435cbaf4ebcac98eedee-106ebc4b0ddd4acc824ae37042907a89 on tablet 106ebc4b0ddd4acc824ae37042907a89 with peer 29b035434087435cbaf4ebcac98eedee
I20240502 08:18:37.416642 18145 tablet_bootstrap.cc:492] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.417126 18145 tablet_bootstrap.cc:492] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:18:37.417382 18145 ts_tablet_manager.cc:1367] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.109s user 0.051s sys 0.008s
I20240502 08:18:37.418165 18145 raft_consensus.cc:348] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:37.418411 18145 raft_consensus.cc:729] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:37.418716 18145 consensus_queue.cc:260] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:37.419440 18145 ts_tablet_manager.cc:1398] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:18:37.420318 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-3c37ff40706043e9adc4da4a6da1947a received from {username='slave'} at 127.6.5.6:55577
I20240502 08:18:37.420436 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-3c37ff40706043e9adc4da4a6da1947a on tablet 3c37ff40706043e9adc4da4a6da1947a with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:37.434923 18173 tablet_copy_client.cc:308] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:18:37.435513 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet ab808742ce8746ae8e9b8fbff435e7be from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:55577)
I20240502 08:18:37.435640 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet ab808742ce8746ae8e9b8fbff435e7be from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:55577: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-ab808742ce8746ae8e9b8fbff435e7be
I20240502 08:18:37.438448 12751 tablet_copy_source_session.cc:205] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:37.440832 18173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab808742ce8746ae8e9b8fbff435e7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:37.443235 18171 tablet_copy_client.cc:308] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.6:44983
I20240502 08:18:37.444110 17729 raft_consensus.cc:2932] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:37.446717 15642 tablet_copy_service.cc:140] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Received BeginTabletCopySession request for tablet e9081ea0b98443d38f42c14cd626f0d0 from peer 9333cb10227b4801b4fa7c24e22e131b ({username='slave'} at 127.6.5.7:47489)
I20240502 08:18:37.446966 15642 tablet_copy_service.cc:161] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Beginning new tablet copy session on tablet e9081ea0b98443d38f42c14cd626f0d0 from peer 9333cb10227b4801b4fa7c24e22e131b at {username='slave'} at 127.6.5.7:47489: session id = 9333cb10227b4801b4fa7c24e22e131b-e9081ea0b98443d38f42c14cd626f0d0
I20240502 08:18:37.448751 18173 tablet_copy_client.cc:783] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:37.449079 18173 tablet_copy_client.cc:647] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:37.450507 18173 tablet_copy_client.cc:523] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:37.450835 15642 tablet_copy_source_session.cc:205] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:37.453433 18171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9081ea0b98443d38f42c14cd626f0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:37.459385 15488 raft_consensus.cc:2932] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:37.463599 18171 tablet_copy_client.cc:783] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:37.463932 18171 tablet_copy_client.cc:647] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:37.464367 15887 raft_consensus.cc:2932] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:37.469394 12584 catalog_manager.cc:5484] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:37.469887 18173 tablet_bootstrap.cc:492] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:37.497413 18171 tablet_copy_client.cc:523] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:37.501665 18171 tablet_bootstrap.cc:492] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:37.525103 18171 tablet_bootstrap.cc:492] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.525571 18171 tablet_bootstrap.cc:492] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap complete.
I20240502 08:18:37.525828 18171 ts_tablet_manager.cc:1367] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.010s
I20240502 08:18:37.526660 18171 raft_consensus.cc:348] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:37.526921 18171 raft_consensus.cc:729] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:37.527180 18171 consensus_queue.cc:260] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:37.528415 18171 ts_tablet_manager.cc:1398] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:18:37.529392 15642 tablet_copy_service.cc:342] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Request end of tablet copy session 9333cb10227b4801b4fa7c24e22e131b-e9081ea0b98443d38f42c14cd626f0d0 received from {username='slave'} at 127.6.5.7:47489
I20240502 08:18:37.529549 15642 tablet_copy_service.cc:433] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: ending tablet copy session 9333cb10227b4801b4fa7c24e22e131b-e9081ea0b98443d38f42c14cd626f0d0 on tablet e9081ea0b98443d38f42c14cd626f0d0 with peer 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:37.534037 18173 tablet_bootstrap.cc:492] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.534447 18173 tablet_bootstrap.cc:492] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:18:37.534775 18173 ts_tablet_manager.cc:1367] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.065s user 0.034s sys 0.011s
I20240502 08:18:37.535545 18173 raft_consensus.cc:348] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:37.535796 18173 raft_consensus.cc:729] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:37.536027 18173 consensus_queue.cc:260] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:37.536695 18173 ts_tablet_manager.cc:1398] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:37.537840 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-ab808742ce8746ae8e9b8fbff435e7be received from {username='slave'} at 127.6.5.6:55577
I20240502 08:18:37.537938 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-ab808742ce8746ae8e9b8fbff435e7be on tablet ab808742ce8746ae8e9b8fbff435e7be with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:37.547415 18173 ts_tablet_manager.cc:902] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:37.549113 18166 ts_tablet_manager.cc:902] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:37.551039 18173 tablet_copy_client.cc:308] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:18:37.550943 12730 consensus_queue.cc:237] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:37.551645 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet 9a55a182e1f54db698ba8060eea80da0 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:47559)
I20240502 08:18:37.551820 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet 9a55a182e1f54db698ba8060eea80da0 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:47559: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-9a55a182e1f54db698ba8060eea80da0
I20240502 08:18:37.553740 13119 raft_consensus.cc:1261] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:37.553817 18166 tablet_copy_client.cc:308] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:18:37.554261 17967 consensus_queue.cc:1035] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:37.554572 15508 tablet_copy_source_session.cc:205] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:37.554811 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet 7dcaab7ded074ca38dda02c3bdf4c5e9 from peer 29b035434087435cbaf4ebcac98eedee ({username='slave'} at 127.6.5.1:46969)
I20240502 08:18:37.555037 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet 7dcaab7ded074ca38dda02c3bdf4c5e9 from peer 29b035434087435cbaf4ebcac98eedee at {username='slave'} at 127.6.5.1:46969: session id = 29b035434087435cbaf4ebcac98eedee-7dcaab7ded074ca38dda02c3bdf4c5e9
I20240502 08:18:37.555215 12990 raft_consensus.cc:1261] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:37.555763 18173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a55a182e1f54db698ba8060eea80da0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:18:37.559068 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 809af1a2878d4cba9ab2f18c11a621a8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.561045 13142 tablet_copy_source_session.cc:205] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:37.561666 18173 tablet_copy_client.cc:783] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:37.561883 18173 tablet_copy_client.cc:647] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:37.562489 17967 consensus_queue.cc:1035] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:37.563906 18173 tablet_copy_client.cc:523] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:37.565212 18166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dcaab7ded074ca38dda02c3bdf4c5e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:37.571552 18166 tablet_copy_client.cc:783] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:37.571751 18166 tablet_copy_client.cc:647] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:37.573918 18166 tablet_copy_client.cc:523] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:37.574002 12992 raft_consensus.cc:1203] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:37.574478 18173 tablet_bootstrap.cc:492] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:37.577432 18166 tablet_bootstrap.cc:492] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:37.589705 18166 tablet_bootstrap.cc:492] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.590057 18166 tablet_bootstrap.cc:492] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee: Bootstrap complete.
I20240502 08:18:37.590283 18166 ts_tablet_manager.cc:1367] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20240502 08:18:37.590991 18166 raft_consensus.cc:348] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:37.591218 18166 raft_consensus.cc:729] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:37.591437 18166 consensus_queue.cc:260] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:37.591998 18166 ts_tablet_manager.cc:1398] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:37.592837 18173 tablet_bootstrap.cc:492] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.593276 18173 tablet_bootstrap.cc:492] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:18:37.593538 18173 ts_tablet_manager.cc:1367] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.001s
I20240502 08:18:37.593899 13142 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session 29b035434087435cbaf4ebcac98eedee-7dcaab7ded074ca38dda02c3bdf4c5e9 received from {username='slave'} at 127.6.5.1:46969
I20240502 08:18:37.594044 13142 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session 29b035434087435cbaf4ebcac98eedee-7dcaab7ded074ca38dda02c3bdf4c5e9 on tablet 7dcaab7ded074ca38dda02c3bdf4c5e9 with peer 29b035434087435cbaf4ebcac98eedee
I20240502 08:18:37.594314 18173 raft_consensus.cc:348] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:37.594564 18173 raft_consensus.cc:729] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:37.594800 18173 consensus_queue.cc:260] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:37.595502 18173 ts_tablet_manager.cc:1398] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:37.596480 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-9a55a182e1f54db698ba8060eea80da0 received from {username='slave'} at 127.6.5.6:47559
I20240502 08:18:37.596612 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-9a55a182e1f54db698ba8060eea80da0 on tablet 9a55a182e1f54db698ba8060eea80da0 with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:37.605293 18220 ts_tablet_manager.cc:902] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:37.609491 17967 raft_consensus.cc:2932] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:37.613114 13120 raft_consensus.cc:2932] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:37.620440 12587 catalog_manager.cc:5484] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:37.625492 12990 raft_consensus.cc:2932] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:37.631307 18220 tablet_copy_client.cc:308] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:18:37.636790 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet bf504f2522a147c198f99debdfe03d1e from peer db7893bb29a047b3b82cc60bfd8930a3 ({username='slave'} at 127.6.5.5:59601)
I20240502 08:18:37.636938 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet bf504f2522a147c198f99debdfe03d1e from peer db7893bb29a047b3b82cc60bfd8930a3 at {username='slave'} at 127.6.5.5:59601: session id = db7893bb29a047b3b82cc60bfd8930a3-bf504f2522a147c198f99debdfe03d1e
I20240502 08:18:37.643829 13142 tablet_copy_source_session.cc:205] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:37.645630 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf504f2522a147c198f99debdfe03d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:37.654318 18220 tablet_copy_client.cc:783] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:37.654772 18220 tablet_copy_client.cc:647] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:37.663477 18220 tablet_copy_client.cc:523] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:37.667744 18220 tablet_bootstrap.cc:492] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:37.670956 18230 ts_tablet_manager.cc:902] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:37.675639 12730 raft_consensus.cc:1203] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:37.684933 18141 ts_tablet_manager.cc:902] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4: Initiating tablet copy from peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:37.709177 18230 tablet_copy_client.cc:308] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:18:37.710973 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet f031f5fedb904eaf83a4902092280295 from peer 10f0e36203914b85b77e78609ffced98 ({username='slave'} at 127.6.5.8:54349)
I20240502 08:18:37.711131 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet f031f5fedb904eaf83a4902092280295 from peer 10f0e36203914b85b77e78609ffced98 at {username='slave'} at 127.6.5.8:54349: session id = 10f0e36203914b85b77e78609ffced98-f031f5fedb904eaf83a4902092280295
I20240502 08:18:37.714707 13142 tablet_copy_source_session.cc:205] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:37.717583 18230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f031f5fedb904eaf83a4902092280295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:37.726053 15621 raft_consensus.cc:1203] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:37.728180 18230 tablet_copy_client.cc:783] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:37.728513 18230 tablet_copy_client.cc:647] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:37.730580 18230 tablet_copy_client.cc:523] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:37.738312 18230 tablet_bootstrap.cc:492] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:37.741490 18220 tablet_bootstrap.cc:492] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.741956 18220 tablet_bootstrap.cc:492] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap complete.
I20240502 08:18:37.742255 18220 ts_tablet_manager.cc:1367] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.075s user 0.048s sys 0.013s
I20240502 08:18:37.743103 18220 raft_consensus.cc:348] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:37.743376 18220 raft_consensus.cc:729] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:37.743609 18220 consensus_queue.cc:260] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:37.744215 18220 ts_tablet_manager.cc:1398] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240502 08:18:37.745080 13142 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session db7893bb29a047b3b82cc60bfd8930a3-bf504f2522a147c198f99debdfe03d1e received from {username='slave'} at 127.6.5.5:59601
I20240502 08:18:37.745208 13142 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session db7893bb29a047b3b82cc60bfd8930a3-bf504f2522a147c198f99debdfe03d1e on tablet bf504f2522a147c198f99debdfe03d1e with peer db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:37.774097 18141 tablet_copy_client.cc:308] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.7:38851
I20240502 08:18:37.774752 15775 tablet_copy_service.cc:140] P 9333cb10227b4801b4fa7c24e22e131b: Received BeginTabletCopySession request for tablet ea85ef42ac714074a10f2a5ece4ddf8e from peer 60b6e265e7754b7187307d05ef405dd4 ({username='slave'} at 127.6.5.3:53567)
I20240502 08:18:37.774919 15775 tablet_copy_service.cc:161] P 9333cb10227b4801b4fa7c24e22e131b: Beginning new tablet copy session on tablet ea85ef42ac714074a10f2a5ece4ddf8e from peer 60b6e265e7754b7187307d05ef405dd4 at {username='slave'} at 127.6.5.3:53567: session id = 60b6e265e7754b7187307d05ef405dd4-ea85ef42ac714074a10f2a5ece4ddf8e
I20240502 08:18:37.777745 15775 tablet_copy_source_session.cc:205] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:37.780403 18141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea85ef42ac714074a10f2a5ece4ddf8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:37.787869 18141 tablet_copy_client.cc:783] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:37.788154 18141 tablet_copy_client.cc:647] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:37.791428 18230 tablet_bootstrap.cc:492] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.791925 18230 tablet_bootstrap.cc:492] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98: Bootstrap complete.
I20240502 08:18:37.792237 18230 ts_tablet_manager.cc:1367] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.054s user 0.050s sys 0.000s
I20240502 08:18:37.793150 18230 raft_consensus.cc:348] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:37.793440 18230 raft_consensus.cc:729] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:37.794081 18141 tablet_copy_client.cc:523] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:37.794224 18230 consensus_queue.cc:260] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:37.795238 18230 ts_tablet_manager.cc:1398] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 08:18:37.796075 13142 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session 10f0e36203914b85b77e78609ffced98-f031f5fedb904eaf83a4902092280295 received from {username='slave'} at 127.6.5.8:54349
I20240502 08:18:37.796190 13142 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session 10f0e36203914b85b77e78609ffced98-f031f5fedb904eaf83a4902092280295 on tablet f031f5fedb904eaf83a4902092280295 with peer 10f0e36203914b85b77e78609ffced98
I20240502 08:18:37.798821 18141 tablet_bootstrap.cc:492] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:37.813176 18141 tablet_bootstrap.cc:492] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.813511 18141 tablet_bootstrap.cc:492] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4: Bootstrap complete.
I20240502 08:18:37.813706 18141 ts_tablet_manager.cc:1367] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20240502 08:18:37.814488 18141 raft_consensus.cc:348] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:37.814779 18141 raft_consensus.cc:729] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:37.815032 18141 consensus_queue.cc:260] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:37.815608 18141 ts_tablet_manager.cc:1398] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:37.816768 15775 tablet_copy_service.cc:342] P 9333cb10227b4801b4fa7c24e22e131b: Request end of tablet copy session 60b6e265e7754b7187307d05ef405dd4-ea85ef42ac714074a10f2a5ece4ddf8e received from {username='slave'} at 127.6.5.3:53567
I20240502 08:18:37.816892 15775 tablet_copy_service.cc:433] P 9333cb10227b4801b4fa7c24e22e131b: ending tablet copy session 60b6e265e7754b7187307d05ef405dd4-ea85ef42ac714074a10f2a5ece4ddf8e on tablet ea85ef42ac714074a10f2a5ece4ddf8e with peer 60b6e265e7754b7187307d05ef405dd4
I20240502 08:18:37.828863 15621 raft_consensus.cc:1203] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:37.839474 15886 consensus_queue.cc:237] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:37.842034 15754 raft_consensus.cc:1261] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:37.842233 15818 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Couldn't send request to peer 6a0b05f3250d415590b0edf6b44fbe1f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b7190137f2949de8df2cdf447802db1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.842635 18150 consensus_queue.cc:1035] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:37.842993 15621 raft_consensus.cc:1261] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:37.843533 18150 consensus_queue.cc:1035] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:37.865118 18150 raft_consensus.cc:2932] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:37.869997 15754 raft_consensus.cc:2932] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:37.871619 15622 raft_consensus.cc:2932] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:37.874138 12584 catalog_manager.cc:5484] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:37.878408 15754 raft_consensus.cc:1203] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:37.883721 18263 ts_tablet_manager.cc:902] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:37.887913 18263 tablet_copy_client.cc:308] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.6:44983
I20240502 08:18:37.888662 15642 tablet_copy_service.cc:140] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Received BeginTabletCopySession request for tablet 8774c037b901461684d96460e663d17e from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:42935)
I20240502 08:18:37.888849 15642 tablet_copy_service.cc:161] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Beginning new tablet copy session on tablet 8774c037b901461684d96460e663d17e from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:42935: session id = e1dd14f7ca764a1490df419f13f2804a-8774c037b901461684d96460e663d17e
I20240502 08:18:37.891630 15642 tablet_copy_source_session.cc:205] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:37.893407 18263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8774c037b901461684d96460e663d17e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:37.899183 18263 tablet_copy_client.cc:783] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:37.899473 18263 tablet_copy_client.cc:647] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:37.901332 18263 tablet_copy_client.cc:523] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:37.904628 18263 tablet_bootstrap.cc:492] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:37.911140 18263 tablet_bootstrap.cc:492] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.911471 18263 tablet_bootstrap.cc:492] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:18:37.911672 18263 ts_tablet_manager.cc:1367] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240502 08:18:37.912268 18263 raft_consensus.cc:348] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:37.912494 18263 raft_consensus.cc:729] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:37.912772 18263 consensus_queue.cc:260] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:37.913287 18263 ts_tablet_manager.cc:1398] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:37.913988 15642 tablet_copy_service.cc:342] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-8774c037b901461684d96460e663d17e received from {username='slave'} at 127.6.5.2:42935
I20240502 08:18:37.914108 15642 tablet_copy_service.cc:433] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-8774c037b901461684d96460e663d17e on tablet 8774c037b901461684d96460e663d17e with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:18:37.928154 13119 consensus_queue.cc:237] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:37.930228 12992 raft_consensus.cc:1261] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:37.930245 12730 raft_consensus.cc:1261] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:37.930532 13056 consensus_peers.cc:485] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Couldn't send request to peer db7893bb29a047b3b82cc60bfd8930a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db051ed0438742319f741a565834b644. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.930819 18283 consensus_queue.cc:1035] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:37.931129 18067 consensus_queue.cc:1035] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:37.933633 18067 raft_consensus.cc:2932] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:37.934140 12992 raft_consensus.cc:2932] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:37.939590 12587 catalog_manager.cc:5484] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:37.943464 12727 raft_consensus.cc:2932] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:37.945586 12730 raft_consensus.cc:1203] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:37.979207 15621 raft_consensus.cc:1203] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:37.985672 15754 consensus_queue.cc:237] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:37.987970 15621 raft_consensus.cc:1261] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:37.988025 15886 raft_consensus.cc:1261] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:37.988391 15688 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 156838b04be24853897a4fabb5916029. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.988617 17275 consensus_queue.cc:1035] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:37.988974 18304 consensus_queue.cc:1035] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:37.993013 17275 raft_consensus.cc:2932] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:37.993468 15621 raft_consensus.cc:2932] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:37.997758 15886 raft_consensus.cc:2932] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:37.998518 12583 catalog_manager.cc:5484] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:38.044763 15488 consensus_queue.cc:237] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:38.046849 15754 raft_consensus.cc:1261] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:38.047092 15419 consensus_peers.cc:485] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): Couldn't send request to peer 60b6e265e7754b7187307d05ef405dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95d1c7bd2c424900b399fac3fdb5c9ac. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:38.047127 15621 raft_consensus.cc:1261] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:38.047506 17735 consensus_queue.cc:1035] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:38.048036 17735 consensus_queue.cc:1035] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:38.059592 18302 raft_consensus.cc:2932] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:38.060076 15621 raft_consensus.cc:2932] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:38.066007 12587 catalog_manager.cc:5484] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:38.067019 15755 raft_consensus.cc:2932] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:38.068687 12992 raft_consensus.cc:1203] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:38.123443 15886 consensus_queue.cc:237] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:38.125718 15755 raft_consensus.cc:1261] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:38.125872 15818 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Couldn't send request to peer 6a0b05f3250d415590b0edf6b44fbe1f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad1bea423b354df38e9e0100c1560e08. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:38.125895 15621 raft_consensus.cc:1261] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:38.126199 18150 consensus_queue.cc:1035] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:38.126734 18257 consensus_queue.cc:1035] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:38.129485 18150 raft_consensus.cc:2932] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:38.130218 15621 raft_consensus.cc:2932] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:38.134941 15755 raft_consensus.cc:2932] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:38.137198 12583 catalog_manager.cc:5484] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:38.156595 18350 ts_tablet_manager.cc:902] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:38.158223 18350 tablet_copy_client.cc:308] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:18:38.159655 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet 809af1a2878d4cba9ab2f18c11a621a8 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:55577)
I20240502 08:18:38.159801 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet 809af1a2878d4cba9ab2f18c11a621a8 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:55577: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-809af1a2878d4cba9ab2f18c11a621a8
I20240502 08:18:38.162390 12751 tablet_copy_source_session.cc:205] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:38.164003 18350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809af1a2878d4cba9ab2f18c11a621a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:38.165984 15487 raft_consensus.cc:1203] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:38.169772 18350 tablet_copy_client.cc:783] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:38.170017 18350 tablet_copy_client.cc:647] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:38.173517 18350 tablet_copy_client.cc:523] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:38.176450 18350 tablet_bootstrap.cc:492] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:38.214027 13119 consensus_queue.cc:237] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:38.220580 18350 tablet_bootstrap.cc:492] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:38.221407 18350 tablet_bootstrap.cc:492] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:18:38.221676 18350 ts_tablet_manager.cc:1367] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.045s user 0.036s sys 0.008s
I20240502 08:18:38.222527 18350 raft_consensus.cc:348] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:38.222781 18350 raft_consensus.cc:729] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:38.223004 18350 consensus_queue.cc:260] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:38.223677 18350 ts_tablet_manager.cc:1398] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:38.223940 12859 raft_consensus.cc:1261] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:38.224027 12727 raft_consensus.cc:1261] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:38.225378 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-809af1a2878d4cba9ab2f18c11a621a8 received from {username='slave'} at 127.6.5.6:55577
I20240502 08:18:38.225497 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-809af1a2878d4cba9ab2f18c11a621a8 on tablet 809af1a2878d4cba9ab2f18c11a621a8 with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:38.225780 18067 consensus_queue.cc:1035] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:38.226308 18067 consensus_queue.cc:1035] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:18:38.228224 13056 consensus_peers.cc:485] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Couldn't send request to peer db7893bb29a047b3b82cc60bfd8930a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91f185fbf27049a7992daa1d905e5575. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:38.263370 18369 ts_tablet_manager.cc:902] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f: Initiating tablet copy from peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:38.263564 18283 raft_consensus.cc:2932] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:38.266041 15887 raft_consensus.cc:1203] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:38.267429 12859 raft_consensus.cc:2932] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:38.270859 12727 raft_consensus.cc:2932] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:38.280637 12583 catalog_manager.cc:5484] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:38.307574 18369 tablet_copy_client.cc:308] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.8:46303
I20240502 08:18:38.308686 15907 tablet_copy_service.cc:140] P 10f0e36203914b85b77e78609ffced98: Received BeginTabletCopySession request for tablet 1b7190137f2949de8df2cdf447802db1 from peer 6a0b05f3250d415590b0edf6b44fbe1f ({username='slave'} at 127.6.5.4:32771)
I20240502 08:18:38.308941 15907 tablet_copy_service.cc:161] P 10f0e36203914b85b77e78609ffced98: Beginning new tablet copy session on tablet 1b7190137f2949de8df2cdf447802db1 from peer 6a0b05f3250d415590b0edf6b44fbe1f at {username='slave'} at 127.6.5.4:32771: session id = 6a0b05f3250d415590b0edf6b44fbe1f-1b7190137f2949de8df2cdf447802db1
I20240502 08:18:38.313053 15907 tablet_copy_source_session.cc:205] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:38.315078 18369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7190137f2949de8df2cdf447802db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:38.321718 18369 tablet_copy_client.cc:783] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:38.321990 18369 tablet_copy_client.cc:647] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:38.324338 18369 tablet_copy_client.cc:523] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:38.328467 18369 tablet_bootstrap.cc:492] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:38.342922 18369 tablet_bootstrap.cc:492] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:38.345814 18369 tablet_bootstrap.cc:492] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap complete.
I20240502 08:18:38.346081 18369 ts_tablet_manager.cc:1367] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.010s
I20240502 08:18:38.346818 18369 raft_consensus.cc:348] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:38.347038 18369 raft_consensus.cc:729] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:38.347373 18369 consensus_queue.cc:260] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:38.348227 18369 ts_tablet_manager.cc:1398] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:38.349267 15907 tablet_copy_service.cc:342] P 10f0e36203914b85b77e78609ffced98: Request end of tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-1b7190137f2949de8df2cdf447802db1 received from {username='slave'} at 127.6.5.4:32771
I20240502 08:18:38.349432 15907 tablet_copy_service.cc:433] P 10f0e36203914b85b77e78609ffced98: ending tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-1b7190137f2949de8df2cdf447802db1 on tablet 1b7190137f2949de8df2cdf447802db1 with peer 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:18:38.379513 12859 raft_consensus.cc:1203] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:38.391243 15488 consensus_queue.cc:237] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:38.393776 15887 raft_consensus.cc:1261] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:38.394531 13119 raft_consensus.cc:1261] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:38.394421 18302 consensus_queue.cc:1035] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:38.395035 17735 consensus_queue.cc:1035] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:38.397243 15419 consensus_peers.cc:485] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ead2e2b9f05a4a729bcba36dcec35a0a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:38.406688 18302 raft_consensus.cc:2932] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:38.407363 13119 raft_consensus.cc:2932] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:38.414134 15887 raft_consensus.cc:2932] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:38.414433 12584 catalog_manager.cc:5484] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:38.512907 18411 ts_tablet_manager.cc:902] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:38.514725 18411 tablet_copy_client.cc:308] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:18:38.515659 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet db051ed0438742319f741a565834b644 from peer db7893bb29a047b3b82cc60bfd8930a3 ({username='slave'} at 127.6.5.5:59601)
I20240502 08:18:38.515826 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet db051ed0438742319f741a565834b644 from peer db7893bb29a047b3b82cc60bfd8930a3 at {username='slave'} at 127.6.5.5:59601: session id = db7893bb29a047b3b82cc60bfd8930a3-db051ed0438742319f741a565834b644
I20240502 08:18:38.518257 13142 tablet_copy_source_session.cc:205] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:38.519970 18411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db051ed0438742319f741a565834b644. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:38.525483 18411 tablet_copy_client.cc:783] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:38.525743 18411 tablet_copy_client.cc:647] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:38.527920 18411 tablet_copy_client.cc:523] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:38.531203 18411 tablet_bootstrap.cc:492] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:38.583814 18411 tablet_bootstrap.cc:492] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:38.584266 18411 tablet_bootstrap.cc:492] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap complete.
I20240502 08:18:38.584548 18411 ts_tablet_manager.cc:1367] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.053s user 0.041s sys 0.004s
I20240502 08:18:38.585449 18411 raft_consensus.cc:348] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:38.585697 18411 raft_consensus.cc:729] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:38.585937 18411 consensus_queue.cc:260] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:38.586553 18411 ts_tablet_manager.cc:1398] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:38.587205 13142 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session db7893bb29a047b3b82cc60bfd8930a3-db051ed0438742319f741a565834b644 received from {username='slave'} at 127.6.5.5:59601
I20240502 08:18:38.587355 13142 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session db7893bb29a047b3b82cc60bfd8930a3-db051ed0438742319f741a565834b644 on tablet db051ed0438742319f741a565834b644 with peer db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:38.611721 15622 raft_consensus.cc:1203] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:38.622712 18432 ts_tablet_manager.cc:902] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
W20240502 08:18:38.685336 12665 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.8:46303, user_credentials={real_user=slave}} blocked reactor thread for 52529.6us
I20240502 08:18:38.693780 18437 ts_tablet_manager.cc:902] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:38.700572 18411 ts_tablet_manager.cc:902] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:38.712771 18369 ts_tablet_manager.cc:902] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f: Initiating tablet copy from peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:38.721561 18432 tablet_copy_client.cc:308] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.7:38851
I20240502 08:18:38.722208 15775 tablet_copy_service.cc:140] P 9333cb10227b4801b4fa7c24e22e131b: Received BeginTabletCopySession request for tablet 156838b04be24853897a4fabb5916029 from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:37259)
I20240502 08:18:38.722373 15775 tablet_copy_service.cc:161] P 9333cb10227b4801b4fa7c24e22e131b: Beginning new tablet copy session on tablet 156838b04be24853897a4fabb5916029 from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:37259: session id = e1dd14f7ca764a1490df419f13f2804a-156838b04be24853897a4fabb5916029
I20240502 08:18:38.725131 15775 tablet_copy_source_session.cc:205] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:38.726382 18432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156838b04be24853897a4fabb5916029. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:38.732528 18432 tablet_copy_client.cc:783] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:38.732745 18432 tablet_copy_client.cc:647] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:38.751899 18432 tablet_copy_client.cc:523] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:38.752419 13120 raft_consensus.cc:1203] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:38.754295 18369 tablet_copy_client.cc:308] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.8:46303
I20240502 08:18:38.755012 15907 tablet_copy_service.cc:140] P 10f0e36203914b85b77e78609ffced98: Received BeginTabletCopySession request for tablet ad1bea423b354df38e9e0100c1560e08 from peer 6a0b05f3250d415590b0edf6b44fbe1f ({username='slave'} at 127.6.5.4:32771)
I20240502 08:18:38.755200 15907 tablet_copy_service.cc:161] P 10f0e36203914b85b77e78609ffced98: Beginning new tablet copy session on tablet ad1bea423b354df38e9e0100c1560e08 from peer 6a0b05f3250d415590b0edf6b44fbe1f at {username='slave'} at 127.6.5.4:32771: session id = 6a0b05f3250d415590b0edf6b44fbe1f-ad1bea423b354df38e9e0100c1560e08
I20240502 08:18:38.755299 18432 tablet_bootstrap.cc:492] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:38.756879 18411 tablet_copy_client.cc:308] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:18:38.757870 15907 tablet_copy_source_session.cc:205] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:38.758577 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet 91f185fbf27049a7992daa1d905e5575 from peer db7893bb29a047b3b82cc60bfd8930a3 ({username='slave'} at 127.6.5.5:59601)
I20240502 08:18:38.758728 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet 91f185fbf27049a7992daa1d905e5575 from peer db7893bb29a047b3b82cc60bfd8930a3 at {username='slave'} at 127.6.5.5:59601: session id = db7893bb29a047b3b82cc60bfd8930a3-91f185fbf27049a7992daa1d905e5575
I20240502 08:18:38.759680 18369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1bea423b354df38e9e0100c1560e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:38.761732 13142 tablet_copy_source_session.cc:205] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:38.769491 18411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f185fbf27049a7992daa1d905e5575. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:38.777694 18369 tablet_copy_client.cc:783] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:38.777981 18369 tablet_copy_client.cc:647] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:38.783982 18369 tablet_copy_client.cc:523] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:38.791790 18432 tablet_bootstrap.cc:492] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:38.792187 18432 tablet_bootstrap.cc:492] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:18:38.792460 18432 ts_tablet_manager.cc:1367] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.003s
I20240502 08:18:38.793239 18432 raft_consensus.cc:348] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:38.793474 18432 raft_consensus.cc:729] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:38.793740 18432 consensus_queue.cc:260] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:38.794342 18432 ts_tablet_manager.cc:1398] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:18:38.795953 15775 tablet_copy_service.cc:342] P 9333cb10227b4801b4fa7c24e22e131b: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-156838b04be24853897a4fabb5916029 received from {username='slave'} at 127.6.5.2:37259
I20240502 08:18:38.796108 15775 tablet_copy_service.cc:433] P 9333cb10227b4801b4fa7c24e22e131b: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-156838b04be24853897a4fabb5916029 on tablet 156838b04be24853897a4fabb5916029 with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:18:38.797581 18432 ts_tablet_manager.cc:902] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:38.805756 18369 tablet_bootstrap.cc:492] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:38.823724 18437 tablet_copy_client.cc:308] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:18:38.824352 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet 95d1c7bd2c424900b399fac3fdb5c9ac from peer 60b6e265e7754b7187307d05ef405dd4 ({username='slave'} at 127.6.5.3:43031)
I20240502 08:18:38.824519 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet 95d1c7bd2c424900b399fac3fdb5c9ac from peer 60b6e265e7754b7187307d05ef405dd4 at {username='slave'} at 127.6.5.3:43031: session id = 60b6e265e7754b7187307d05ef405dd4-95d1c7bd2c424900b399fac3fdb5c9ac
I20240502 08:18:38.827214 15508 tablet_copy_source_session.cc:205] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:38.836879 18437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d1c7bd2c424900b399fac3fdb5c9ac. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:18:38.839488 15554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.1:33183, user_credentials={real_user=slave}} blocked reactor thread for 58082.4us
I20240502 08:18:38.850556 18411 tablet_copy_client.cc:783] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:38.852270 18411 tablet_copy_client.cc:647] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:38.856029 18432 tablet_copy_client.cc:308] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:18:38.856637 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet ead2e2b9f05a4a729bcba36dcec35a0a from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:60127)
I20240502 08:18:38.856806 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet ead2e2b9f05a4a729bcba36dcec35a0a from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:60127: session id = e1dd14f7ca764a1490df419f13f2804a-ead2e2b9f05a4a729bcba36dcec35a0a
I20240502 08:18:38.859618 15508 tablet_copy_source_session.cc:205] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:38.860652 18369 tablet_bootstrap.cc:492] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:38.860666 18432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead2e2b9f05a4a729bcba36dcec35a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:38.861055 18369 tablet_bootstrap.cc:492] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap complete.
I20240502 08:18:38.861331 18369 ts_tablet_manager.cc:1367] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.019s
I20240502 08:18:38.862092 18369 raft_consensus.cc:348] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:38.862381 18369 raft_consensus.cc:729] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:38.862633 18369 consensus_queue.cc:260] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:38.863219 18369 ts_tablet_manager.cc:1398] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:18:38.863459 18411 tablet_copy_client.cc:523] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:38.868736 15907 tablet_copy_service.cc:342] P 10f0e36203914b85b77e78609ffced98: Request end of tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-ad1bea423b354df38e9e0100c1560e08 received from {username='slave'} at 127.6.5.4:32771
I20240502 08:18:38.868877 15907 tablet_copy_service.cc:433] P 10f0e36203914b85b77e78609ffced98: ending tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-ad1bea423b354df38e9e0100c1560e08 on tablet ad1bea423b354df38e9e0100c1560e08 with peer 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:18:38.878567 18411 tablet_bootstrap.cc:492] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:38.878701 18437 tablet_copy_client.cc:783] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:38.878943 18437 tablet_copy_client.cc:647] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:38.881744 18432 tablet_copy_client.cc:783] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:38.882002 18432 tablet_copy_client.cc:647] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:38.883889 18437 tablet_copy_client.cc:523] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:38.883949 18432 tablet_copy_client.cc:523] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:38.890713 18437 tablet_bootstrap.cc:492] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:38.891589 18432 tablet_bootstrap.cc:492] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:38.906118 18432 tablet_bootstrap.cc:492] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:38.906772 18432 tablet_bootstrap.cc:492] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:18:38.907430 18432 ts_tablet_manager.cc:1367] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240502 08:18:38.908497 18432 raft_consensus.cc:348] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:38.909667 18432 raft_consensus.cc:729] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:38.910085 18432 consensus_queue.cc:260] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:38.911139 18432 ts_tablet_manager.cc:1398] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:38.914084 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-ead2e2b9f05a4a729bcba36dcec35a0a received from {username='slave'} at 127.6.5.2:60127
I20240502 08:18:38.914213 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-ead2e2b9f05a4a729bcba36dcec35a0a on tablet ead2e2b9f05a4a729bcba36dcec35a0a with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:18:38.915576 18437 tablet_bootstrap.cc:492] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:38.915935 18437 tablet_bootstrap.cc:492] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4: Bootstrap complete.
I20240502 08:18:38.916170 18437 ts_tablet_manager.cc:1367] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.001s
I20240502 08:18:38.936463 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:38.941411 18437 raft_consensus.cc:348] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:38.941658 18437 raft_consensus.cc:729] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:38.946858 18437 consensus_queue.cc:260] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:38.947561 18437 ts_tablet_manager.cc:1398] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240502 08:18:38.950789 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session 60b6e265e7754b7187307d05ef405dd4-95d1c7bd2c424900b399fac3fdb5c9ac received from {username='slave'} at 127.6.5.3:43031
I20240502 08:18:38.950909 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session 60b6e265e7754b7187307d05ef405dd4-95d1c7bd2c424900b399fac3fdb5c9ac on tablet 95d1c7bd2c424900b399fac3fdb5c9ac with peer 60b6e265e7754b7187307d05ef405dd4
I20240502 08:18:38.966655 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:39.001801 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:39.015724 18411 tablet_bootstrap.cc:492] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:39.016199 18411 tablet_bootstrap.cc:492] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap complete.
I20240502 08:18:39.016505 18411 ts_tablet_manager.cc:1367] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.138s user 0.045s sys 0.004s
I20240502 08:18:39.017285 18411 raft_consensus.cc:348] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:39.017581 18411 raft_consensus.cc:729] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:39.017865 18411 consensus_queue.cc:260] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:39.021029 18411 ts_tablet_manager.cc:1398] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240502 08:18:39.021661 13142 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session db7893bb29a047b3b82cc60bfd8930a3-91f185fbf27049a7992daa1d905e5575 received from {username='slave'} at 127.6.5.5:59601
I20240502 08:18:39.021768 13142 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session db7893bb29a047b3b82cc60bfd8930a3-91f185fbf27049a7992daa1d905e5575 on tablet 91f185fbf27049a7992daa1d905e5575 with peer db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:39.034883 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:39.042169 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:39.081570 15488 raft_consensus.cc:1203] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:39.096402 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:39.126724 12862 raft_consensus.cc:1203] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:39.138149 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:39.141535 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:39.148885 12992 raft_consensus.cc:1203] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:39.207202 12862 raft_consensus.cc:1203] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:39.214263 13120 raft_consensus.cc:1203] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:39.226418 15488 raft_consensus.cc:1203] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:39.541014 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:39.561944 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:39.587545 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:39.605583 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:39.618362 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:39.620343 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:39.640261 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:39.674109 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:39.858703 15488 consensus_queue.cc:237] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:39.860421 12862 raft_consensus.cc:1261] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:39.860832 13119 raft_consensus.cc:1261] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:39.861335 18302 consensus_queue.cc:1035] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:39.861704 17577 consensus_queue.cc:1035] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:39.863895 17577 raft_consensus.cc:2932] T ead2e2b9f05a4a729bcba36dcec35a0a P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:39.864885 13119 raft_consensus.cc:2932] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:39.867370 12862 raft_consensus.cc:2932] T ead2e2b9f05a4a729bcba36dcec35a0a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:39.871461 12583 catalog_manager.cc:5484] T ead2e2b9f05a4a729bcba36dcec35a0a P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:39.885768 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet ead2e2b9f05a4a729bcba36dcec35a0a with delete_type TABLET_DATA_TOMBSTONED (TS 10f0e36203914b85b77e78609ffced98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:39.886483 18588 tablet_replica.cc:330] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:39.886725 18588 raft_consensus.cc:2229] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:39.886953 18588 raft_consensus.cc:2258] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:39.887639 18588 ts_tablet_manager.cc:1875] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:39.895136 18588 ts_tablet_manager.cc:1888] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:39.895248 18588 log.cc:1196] T ead2e2b9f05a4a729bcba36dcec35a0a P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/ead2e2b9f05a4a729bcba36dcec35a0a
I20240502 08:18:39.895877 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet ead2e2b9f05a4a729bcba36dcec35a0a (table rebalancer_extra_table_26 [id=602cdebb879f43d3b1a9666d8bd7cddc]) successfully deleted
I20240502 08:18:40.111639 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:40.173316 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:40.181766 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:40.213975 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:40.218820 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:40.240358 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:40.240777 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:40.257107 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:40.437033 15886 consensus_queue.cc:237] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:40.439087 13119 raft_consensus.cc:1261] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:40.439728 15622 raft_consensus.cc:1261] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:40.440302 18554 consensus_queue.cc:1035] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:40.440837 18554 consensus_queue.cc:1035] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:40.444684 18257 raft_consensus.cc:2932] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:40.449721 15620 raft_consensus.cc:2932] T ad1bea423b354df38e9e0100c1560e08 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:40.451166 13119 raft_consensus.cc:2932] T ad1bea423b354df38e9e0100c1560e08 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:40.450335 12580 catalog_manager.cc:5484] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:40.474148 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1bea423b354df38e9e0100c1560e08 with delete_type TABLET_DATA_TOMBSTONED (TS 9333cb10227b4801b4fa7c24e22e131b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:40.474876 18673 tablet_replica.cc:330] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:40.475147 18673 raft_consensus.cc:2229] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:40.475692 18673 raft_consensus.cc:2258] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:40.476428 18673 ts_tablet_manager.cc:1875] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:40.482926 18673 ts_tablet_manager.cc:1888] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:40.483094 18673 log.cc:1196] T ad1bea423b354df38e9e0100c1560e08 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/ad1bea423b354df38e9e0100c1560e08
I20240502 08:18:40.483876 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet ad1bea423b354df38e9e0100c1560e08 (table rebalancer_extra_table_10 [id=e77588602bd34d11a274be948cc9d52a]) successfully deleted
I20240502 08:18:40.744431 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:40.759994 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:40.799840 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:40.803426 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:40.875702 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:40.881928 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:40.933898 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:40.973598 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:41.113157 13119 consensus_queue.cc:237] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:41.115528 12730 raft_consensus.cc:1261] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:41.116070 15755 raft_consensus.cc:1261] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:41.116371 18494 consensus_queue.cc:1035] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:41.117173 18283 consensus_queue.cc:1035] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:41.119711 18283 raft_consensus.cc:2932] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:41.120172 12730 raft_consensus.cc:2932] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:41.123690 15755 raft_consensus.cc:2932] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:41.127410 12588 catalog_manager.cc:5484] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:41.143610 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dcaab7ded074ca38dda02c3bdf4c5e9 with delete_type TABLET_DATA_TOMBSTONED (TS 10f0e36203914b85b77e78609ffced98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:41.144340 18759 tablet_replica.cc:330] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:41.144557 18759 raft_consensus.cc:2229] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:41.144789 18759 raft_consensus.cc:2258] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:41.145443 18759 ts_tablet_manager.cc:1875] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:41.151454 18759 ts_tablet_manager.cc:1888] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:41.151573 18759 log.cc:1196] T 7dcaab7ded074ca38dda02c3bdf4c5e9 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/7dcaab7ded074ca38dda02c3bdf4c5e9
I20240502 08:18:41.152240 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 7dcaab7ded074ca38dda02c3bdf4c5e9 (table rebalancer_extra_table_44 [id=6eff13bc8a8247babea96b8f0537a4cf]) successfully deleted
I20240502 08:18:41.465790 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:41.471633 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:41.472868 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:41.509145 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:41.538676 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:41.547731 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:41.593286 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:41.627676 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:41.740487 15620 consensus_queue.cc:237] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:41.742496 12861 raft_consensus.cc:1261] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:41.742604 15754 raft_consensus.cc:1261] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:41.743295 17729 consensus_queue.cc:1035] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:41.743633 18553 consensus_queue.cc:1035] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:41.759356 18834 raft_consensus.cc:2932] T e9081ea0b98443d38f42c14cd626f0d0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:41.760107 15754 raft_consensus.cc:2932] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:41.761651 12861 raft_consensus.cc:2932] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:41.766206 12584 catalog_manager.cc:5484] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:41.778868 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet e9081ea0b98443d38f42c14cd626f0d0 with delete_type TABLET_DATA_TOMBSTONED (TS db7893bb29a047b3b82cc60bfd8930a3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:41.779515 18841 tablet_replica.cc:330] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:41.779706 18841 raft_consensus.cc:2229] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:41.779940 18841 raft_consensus.cc:2258] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:41.780473 18841 ts_tablet_manager.cc:1875] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:41.785667 18841 ts_tablet_manager.cc:1888] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:41.785864 18841 log.cc:1196] T e9081ea0b98443d38f42c14cd626f0d0 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/e9081ea0b98443d38f42c14cd626f0d0
I20240502 08:18:41.786798 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet e9081ea0b98443d38f42c14cd626f0d0 (table rebalancer_extra_table_18 [id=320abaf9751a4925a5200feff7f4eacc]) successfully deleted
I20240502 08:18:42.038744 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:42.045130 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:42.067624 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:42.068042 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:42.109395 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:42.141085 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:18:42.199892 15819 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.5:44131, user_credentials={real_user=slave}} blocked reactor thread for 81206us
W20240502 08:18:42.209177 13058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.6:44983, user_credentials={real_user=slave}} blocked reactor thread for 108914us
I20240502 08:18:42.239116 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:42.256923 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:42.381107 13119 consensus_queue.cc:237] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:42.383030 15488 raft_consensus.cc:1261] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:42.383065 12990 raft_consensus.cc:1261] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:42.383754 18494 consensus_queue.cc:1035] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:42.384276 18900 consensus_queue.cc:1035] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:42.386874 18900 raft_consensus.cc:2932] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:42.391198 12990 raft_consensus.cc:2932] T bf504f2522a147c198f99debdfe03d1e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:42.392204 12584 catalog_manager.cc:5484] T bf504f2522a147c198f99debdfe03d1e P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:42.399063 15488 raft_consensus.cc:2932] T bf504f2522a147c198f99debdfe03d1e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:42.402513 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet bf504f2522a147c198f99debdfe03d1e with delete_type TABLET_DATA_TOMBSTONED (TS e1dd14f7ca764a1490df419f13f2804a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:42.409281 18927 tablet_replica.cc:330] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:42.409631 18927 raft_consensus.cc:2229] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:42.409854 18927 raft_consensus.cc:2258] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:42.411285 18927 ts_tablet_manager.cc:1875] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:42.418762 18927 ts_tablet_manager.cc:1888] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:18:42.418862 18927 log.cc:1196] T bf504f2522a147c198f99debdfe03d1e P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/bf504f2522a147c198f99debdfe03d1e
I20240502 08:18:42.419513 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet bf504f2522a147c198f99debdfe03d1e (table rebalance_test_table_3 [id=f32118c718584379bfc02c6d1b29d73e]) successfully deleted
I20240502 08:18:42.648133 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:42.683377 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:42.705427 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:42.711592 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:42.758643 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:42.764827 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:42.880489 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:42.887214 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:42.975827 15887 consensus_queue.cc:237] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:42.977691 13119 raft_consensus.cc:1261] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:42.977870 15619 raft_consensus.cc:1261] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:42.978188 18901 consensus_queue.cc:1035] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:42.978505 18554 consensus_queue.cc:1035] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:42.981042 18554 raft_consensus.cc:2932] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:42.981891 13119 raft_consensus.cc:2932] T 1b7190137f2949de8df2cdf447802db1 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:42.981933 15619 raft_consensus.cc:2932] T 1b7190137f2949de8df2cdf447802db1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:42.987107 12588 catalog_manager.cc:5484] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:43.053756 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7190137f2949de8df2cdf447802db1 with delete_type TABLET_DATA_TOMBSTONED (TS 9333cb10227b4801b4fa7c24e22e131b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:43.066900 19017 tablet_replica.cc:330] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:43.067103 19017 raft_consensus.cc:2229] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:43.067381 19017 raft_consensus.cc:2258] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:43.067945 19017 ts_tablet_manager.cc:1875] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:43.073799 19017 ts_tablet_manager.cc:1888] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:43.073988 19017 log.cc:1196] T 1b7190137f2949de8df2cdf447802db1 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/1b7190137f2949de8df2cdf447802db1
I20240502 08:18:43.074790 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 1b7190137f2949de8df2cdf447802db1 (table rebalancer_extra_table_6 [id=f19b8479d1b74f17b2ee9e2117749102]) successfully deleted
I20240502 08:18:43.310271 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:43.345912 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:43.349592 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:43.372767 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:43.431116 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:43.440960 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:43.453673 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:43.545024 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:43.680675 15486 consensus_queue.cc:237] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:43.684971 12990 raft_consensus.cc:1261] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:43.685179 15754 raft_consensus.cc:1261] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:43.685709 18794 consensus_queue.cc:1035] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:43.686043 18302 consensus_queue.cc:1035] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:43.689936 18302 raft_consensus.cc:2932] T b5cd9e39bfc740b78492d369532d0c66 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:43.691042 15753 raft_consensus.cc:2932] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:43.692972 12990 raft_consensus.cc:2932] T b5cd9e39bfc740b78492d369532d0c66 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:43.696707 12579 catalog_manager.cc:5484] T b5cd9e39bfc740b78492d369532d0c66 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index 2 to 3, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:43.701032 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet b5cd9e39bfc740b78492d369532d0c66 with delete_type TABLET_DATA_TOMBSTONED (TS e1dd14f7ca764a1490df419f13f2804a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:43.701627 19088 tablet_replica.cc:330] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:43.701823 19088 raft_consensus.cc:2229] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:43.702023 19088 raft_consensus.cc:2258] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:43.702659 19088 ts_tablet_manager.cc:1875] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:43.708452 19088 ts_tablet_manager.cc:1888] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:43.708588 19088 log.cc:1196] T b5cd9e39bfc740b78492d369532d0c66 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/b5cd9e39bfc740b78492d369532d0c66
I20240502 08:18:43.709252 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet b5cd9e39bfc740b78492d369532d0c66 (table rebalancer_extra_table_14 [id=70609143f5404be1badc9b37bce65793]) successfully deleted
I20240502 08:18:43.957967 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:43.966190 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:43.990761 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:44.019585 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:44.029008 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:44.031834 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:44.046378 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:44.122840 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:44.246397 12730 consensus_queue.cc:237] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:44.248659 15619 raft_consensus.cc:1261] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:44.248725 12861 raft_consensus.cc:1261] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:44.249146 18217 consensus_queue.cc:1035] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:44.249387 18438 consensus_queue.cc:1035] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:44.253015 18217 raft_consensus.cc:2932] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:44.254760 12860 raft_consensus.cc:2932] T ab808742ce8746ae8e9b8fbff435e7be P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:44.255676 15619 raft_consensus.cc:2932] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:44.262446 12579 catalog_manager.cc:5484] T ab808742ce8746ae8e9b8fbff435e7be P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:44.271019 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet ab808742ce8746ae8e9b8fbff435e7be with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:44.276347 19178 tablet_replica.cc:330] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:44.276533 19178 raft_consensus.cc:2229] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:44.276757 19178 raft_consensus.cc:2258] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:44.277336 19178 ts_tablet_manager.cc:1875] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:44.284670 19178 ts_tablet_manager.cc:1888] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:18:44.284850 19178 log.cc:1196] T ab808742ce8746ae8e9b8fbff435e7be P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/ab808742ce8746ae8e9b8fbff435e7be
I20240502 08:18:44.285531 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet ab808742ce8746ae8e9b8fbff435e7be (table rebalance_test_table_9 [id=ab2e0dad5c0341efa8c2321070416a08]) successfully deleted
I20240502 08:18:44.568559 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:44.602396 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:44.613175 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:44.628320 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:44.642527 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:44.685906 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:44.735289 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:44.747634 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:44.877143 12727 consensus_queue.cc:237] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:44.878940 15621 raft_consensus.cc:1261] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:44.878957 12860 raft_consensus.cc:1261] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:44.879388 18438 consensus_queue.cc:1035] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:44.879654 18217 consensus_queue.cc:1035] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:44.889832 18438 raft_consensus.cc:2932] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:44.890283 12861 raft_consensus.cc:2932] T 3c37ff40706043e9adc4da4a6da1947a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:44.890938 15621 raft_consensus.cc:2932] T 3c37ff40706043e9adc4da4a6da1947a P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:44.896169 12580 catalog_manager.cc:5484] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:44.911055 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c37ff40706043e9adc4da4a6da1947a with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:44.914618 19272 tablet_replica.cc:330] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:44.914880 19272 raft_consensus.cc:2229] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:44.915191 19272 raft_consensus.cc:2258] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:44.915915 19272 ts_tablet_manager.cc:1875] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:44.926843 19272 ts_tablet_manager.cc:1888] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:18:44.926947 19272 log.cc:1196] T 3c37ff40706043e9adc4da4a6da1947a P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/3c37ff40706043e9adc4da4a6da1947a
I20240502 08:18:44.927606 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 3c37ff40706043e9adc4da4a6da1947a (table rebalance_test_table_11 [id=02e24b5edb434940bb115fce3e1947fb]) successfully deleted
I20240502 08:18:45.157876 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:45.167429 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:45.168020 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:45.207595 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:45.208302 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:45.211475 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:45.234392 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:45.263864 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:45.413986 15621 consensus_queue.cc:237] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:45.416016 12861 raft_consensus.cc:1261] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:45.416177 15885 raft_consensus.cc:1261] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:45.416499 18553 consensus_queue.cc:1035] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:45.416795 19217 consensus_queue.cc:1035] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:45.420687 18553 raft_consensus.cc:2932] T 8774c037b901461684d96460e663d17e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:45.421202 15885 raft_consensus.cc:2932] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:45.424032 12861 raft_consensus.cc:2932] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:45.427903 12584 catalog_manager.cc:5484] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:45.465351 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet 8774c037b901461684d96460e663d17e with delete_type TABLET_DATA_TOMBSTONED (TS db7893bb29a047b3b82cc60bfd8930a3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46880
I20240502 08:18:45.478819 19352 tablet_replica.cc:330] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:18:45.479171 19352 raft_consensus.cc:2229] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:45.479514 19352 raft_consensus.cc:2258] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:45.480177 19352 ts_tablet_manager.cc:1875] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:45.486482 19352 ts_tablet_manager.cc:1888] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:45.486647 19352 log.cc:1196] T 8774c037b901461684d96460e663d17e P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/8774c037b901461684d96460e663d17e
I20240502 08:18:45.487502 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 8774c037b901461684d96460e663d17e (table rebalancer_extra_table_12 [id=69a26fefc8344774a5308e75413de1d7]) successfully deleted
I20240502 08:18:45.741366 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:45.778376 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:45.782503 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:45.826985 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:45.844267 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:45.855388 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:45.873708 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:45.884300 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:46.066262 15755 consensus_queue.cc:237] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:46.068429 12730 raft_consensus.cc:1261] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:46.068456 15486 raft_consensus.cc:1261] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:46.069016 18725 consensus_queue.cc:1035] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:46.070408 18725 consensus_queue.cc:1035] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:46.072382 18557 raft_consensus.cc:2932] T 106ebc4b0ddd4acc824ae37042907a89 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:46.074187 12730 raft_consensus.cc:2932] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:46.080047 12581 catalog_manager.cc:5484] T 106ebc4b0ddd4acc824ae37042907a89 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:46.082548 15486 raft_consensus.cc:2932] T 106ebc4b0ddd4acc824ae37042907a89 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:46.093063 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 106ebc4b0ddd4acc824ae37042907a89 with delete_type TABLET_DATA_TOMBSTONED (TS 10f0e36203914b85b77e78609ffced98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:46.093717 19437 tablet_replica.cc:330] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:46.093967 19437 raft_consensus.cc:2229] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:46.094246 19437 raft_consensus.cc:2258] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:46.094774 19437 ts_tablet_manager.cc:1875] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:46.099610 19437 ts_tablet_manager.cc:1888] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:46.099767 19437 log.cc:1196] T 106ebc4b0ddd4acc824ae37042907a89 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/106ebc4b0ddd4acc824ae37042907a89
I20240502 08:18:46.100378 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 106ebc4b0ddd4acc824ae37042907a89 (table rebalancer_extra_table_8 [id=4b4d8bad8ac34c61b28c021c55674598]) successfully deleted
I20240502 08:18:46.394134 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:46.398065 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:46.403908 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:46.425599 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:46.439558 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:46.443989 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:46.528781 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:46.556252 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:46.658926 15755 consensus_queue.cc:237] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:46.661285 15621 raft_consensus.cc:1261] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:46.661326 12990 raft_consensus.cc:1261] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:46.661867 19478 consensus_queue.cc:1035] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:46.663408 18725 consensus_queue.cc:1035] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:46.664853 18725 raft_consensus.cc:2932] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:46.669431 12990 raft_consensus.cc:2932] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:46.670209 15621 raft_consensus.cc:2932] T ea85ef42ac714074a10f2a5ece4ddf8e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:46.676918 12582 catalog_manager.cc:5484] T ea85ef42ac714074a10f2a5ece4ddf8e P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 2 to 3, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:46.687929 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet ea85ef42ac714074a10f2a5ece4ddf8e with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:46.702344 19521 tablet_replica.cc:330] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:46.702662 19521 raft_consensus.cc:2229] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:46.702862 19521 raft_consensus.cc:2258] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:46.703359 19521 ts_tablet_manager.cc:1875] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:46.707952 19521 ts_tablet_manager.cc:1888] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:46.708065 19521 log.cc:1196] T ea85ef42ac714074a10f2a5ece4ddf8e P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/ea85ef42ac714074a10f2a5ece4ddf8e
I20240502 08:18:46.708616 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet ea85ef42ac714074a10f2a5ece4ddf8e (table rebalancer_extra_table_56 [id=97336a3aa550477cb0f32e30f0e2a47c]) successfully deleted
I20240502 08:18:46.993455 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:47.000049 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:47.017956 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:47.032744 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:47.081218 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:47.142729 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:47.148983 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:47.174602 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:18:47.187398 12798 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.6:44983, user_credentials={real_user=slave}} blocked reactor thread for 45165.7us
W20240502 08:18:47.232829 12795 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.4:32979, user_credentials={real_user=slave}} blocked reactor thread for 90548.2us
I20240502 08:18:47.356789 15486 consensus_queue.cc:237] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:47.359176 12992 raft_consensus.cc:1261] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:47.359166 15619 raft_consensus.cc:1261] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:47.359740 18794 consensus_queue.cc:1035] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:47.360073 19391 consensus_queue.cc:1035] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:47.363065 19391 raft_consensus.cc:2932] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:47.367599 12992 raft_consensus.cc:2932] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:47.373224 12580 catalog_manager.cc:5484] T 9a55a182e1f54db698ba8060eea80da0 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:47.381251 15619 raft_consensus.cc:2932] T 9a55a182e1f54db698ba8060eea80da0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:47.386186 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a55a182e1f54db698ba8060eea80da0 with delete_type TABLET_DATA_TOMBSTONED (TS 9333cb10227b4801b4fa7c24e22e131b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45134
I20240502 08:18:47.389453 19621 tablet_replica.cc:330] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:18:47.389647 19621 raft_consensus.cc:2229] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:47.389847 19621 raft_consensus.cc:2258] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:47.390391 19621 ts_tablet_manager.cc:1875] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:47.395634 19621 ts_tablet_manager.cc:1888] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:47.395772 19621 log.cc:1196] T 9a55a182e1f54db698ba8060eea80da0 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/9a55a182e1f54db698ba8060eea80da0
I20240502 08:18:47.396561 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 9a55a182e1f54db698ba8060eea80da0 (table rebalancer_extra_table_22 [id=59dbf2adb22b41a4b7b92d1b39c10d56]) successfully deleted
I20240502 08:18:47.640142 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:47.676849 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:47.694742 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:47.711575 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:47.712211 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:47.767398 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:47.774816 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:47.776825 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:47.917243 13119 consensus_queue.cc:237] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:47.919214 12992 raft_consensus.cc:1261] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:47.919296 15486 raft_consensus.cc:1261] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:47.919808 19629 consensus_queue.cc:1035] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:47.920078 19584 consensus_queue.cc:1035] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:47.922744 19584 raft_consensus.cc:2932] T db051ed0438742319f741a565834b644 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:47.923537 15486 raft_consensus.cc:2932] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:47.929476 12584 catalog_manager.cc:5484] T db051ed0438742319f741a565834b644 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:47.934870 12708 tablet_service.cc:1514] Processing DeleteTablet for tablet db051ed0438742319f741a565834b644 with delete_type TABLET_DATA_TOMBSTONED (TS 29b035434087435cbaf4ebcac98eedee not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:47.936816 19696 tablet_replica.cc:330] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:47.937006 12992 raft_consensus.cc:2932] T db051ed0438742319f741a565834b644 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:47.938149 19696 raft_consensus.cc:2229] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:47.938490 19696 raft_consensus.cc:2258] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:47.939249 19696 ts_tablet_manager.cc:1875] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:47.946979 19696 ts_tablet_manager.cc:1888] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:18:47.947144 19696 log.cc:1196] T db051ed0438742319f741a565834b644 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/db051ed0438742319f741a565834b644
I20240502 08:18:47.947935 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet db051ed0438742319f741a565834b644 (table rebalance_test_table_9 [id=ab2e0dad5c0341efa8c2321070416a08]) successfully deleted
I20240502 08:18:48.242405 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:48.329049 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:48.333688 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:48.340854 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:48.383545 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:48.399960 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:48.499276 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:48.501495 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:48.637795 13120 consensus_queue.cc:237] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:48.640300 15887 raft_consensus.cc:1261] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:48.640504 12860 raft_consensus.cc:1261] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:48.640919 19629 consensus_queue.cc:1035] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:48.641989 19629 consensus_queue.cc:1035] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:48.645591 19629 raft_consensus.cc:2932] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:48.651219 12860 raft_consensus.cc:2932] T f031f5fedb904eaf83a4902092280295 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:48.653573 12585 catalog_manager.cc:5484] T f031f5fedb904eaf83a4902092280295 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:48.657874 15885 raft_consensus.cc:2932] T f031f5fedb904eaf83a4902092280295 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:48.690395 12708 tablet_service.cc:1514] Processing DeleteTablet for tablet f031f5fedb904eaf83a4902092280295 with delete_type TABLET_DATA_TOMBSTONED (TS 29b035434087435cbaf4ebcac98eedee not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:48.711799 19795 tablet_replica.cc:330] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:48.712045 19795 raft_consensus.cc:2229] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:48.712378 19795 raft_consensus.cc:2258] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:48.713616 19795 ts_tablet_manager.cc:1875] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:48.721170 19795 ts_tablet_manager.cc:1888] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:18:48.721302 19795 log.cc:1196] T f031f5fedb904eaf83a4902092280295 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/f031f5fedb904eaf83a4902092280295
I20240502 08:18:48.721999 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet f031f5fedb904eaf83a4902092280295 (table rebalance_test_table_7 [id=7ed6dd5b8e8d45edb609b77e58525533]) successfully deleted
W20240502 08:18:48.919073 12926 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.2:36933, user_credentials={real_user=slave}} blocked reactor thread for 46724.7us
I20240502 08:18:49.038220 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:49.040490 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:49.060664 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:49.072810 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:49.087711 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:49.093904 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:49.176362 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:49.176873 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:49.295698 12730 consensus_queue.cc:237] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:49.297904 13120 raft_consensus.cc:1261] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:49.298170 15621 raft_consensus.cc:1261] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:49.298574 18217 consensus_queue.cc:1035] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:49.298828 19592 consensus_queue.cc:1035] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:49.301331 19592 raft_consensus.cc:2932] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:49.302699 13120 raft_consensus.cc:2932] T 809af1a2878d4cba9ab2f18c11a621a8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:49.307157 12584 catalog_manager.cc:5484] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:49.311026 15621 raft_consensus.cc:2932] T 809af1a2878d4cba9ab2f18c11a621a8 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:49.324836 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 809af1a2878d4cba9ab2f18c11a621a8 with delete_type TABLET_DATA_TOMBSTONED (TS 60b6e265e7754b7187307d05ef405dd4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38740
I20240502 08:18:49.338866 19893 tablet_replica.cc:330] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:18:49.339185 19893 raft_consensus.cc:2229] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:49.339480 19893 raft_consensus.cc:2258] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:49.340237 19893 ts_tablet_manager.cc:1875] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:49.347206 19893 ts_tablet_manager.cc:1888] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:18:49.347386 19893 log.cc:1196] T 809af1a2878d4cba9ab2f18c11a621a8 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/809af1a2878d4cba9ab2f18c11a621a8
I20240502 08:18:49.348150 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 809af1a2878d4cba9ab2f18c11a621a8 (table rebalance_test_table_3 [id=f32118c718584379bfc02c6d1b29d73e]) successfully deleted
I20240502 08:18:49.595351 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:49.629967 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:49.637926 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:49.657001 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:49.666813 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:49.693111 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:49.720942 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:49.757818 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:49.909121 15754 consensus_queue.cc:237] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:49.911166 12860 raft_consensus.cc:1261] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:49.911170 15622 raft_consensus.cc:1261] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:49.911763 19948 consensus_queue.cc:1035] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:49.912051 19740 consensus_queue.cc:1035] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:49.914660 19517 raft_consensus.cc:2932] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:49.915169 12860 raft_consensus.cc:2932] T 156838b04be24853897a4fabb5916029 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:49.915242 15620 raft_consensus.cc:2932] T 156838b04be24853897a4fabb5916029 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:49.920475 12580 catalog_manager.cc:5484] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:49.935835 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 156838b04be24853897a4fabb5916029 with delete_type TABLET_DATA_TOMBSTONED (TS 10f0e36203914b85b77e78609ffced98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:49.936652 19975 tablet_replica.cc:330] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:49.936947 19975 raft_consensus.cc:2229] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:49.937242 19975 raft_consensus.cc:2258] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:49.937788 19975 ts_tablet_manager.cc:1875] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:49.943979 19975 ts_tablet_manager.cc:1888] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:49.944087 19975 log.cc:1196] T 156838b04be24853897a4fabb5916029 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/156838b04be24853897a4fabb5916029
I20240502 08:18:49.944761 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 156838b04be24853897a4fabb5916029 (table rebalancer_extra_table_28 [id=b6ad25e45eb542c3ab28cb99d251a5d7]) successfully deleted
I20240502 08:18:50.203933 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:50.225237 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:50.302947 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:50.321982 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:50.367717 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:50.411463 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:50.412863 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:50.444340 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:50.566825 15487 consensus_queue.cc:237] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:50.568807 12989 raft_consensus.cc:1261] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:50.568969 15754 raft_consensus.cc:1261] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:50.569396 19614 consensus_queue.cc:1035] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:50.569648 19990 consensus_queue.cc:1035] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:50.572314 19614 raft_consensus.cc:2932] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:50.572909 12989 raft_consensus.cc:2932] T 95d1c7bd2c424900b399fac3fdb5c9ac P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:50.573207 15754 raft_consensus.cc:2932] T 95d1c7bd2c424900b399fac3fdb5c9ac P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:50.578545 12585 catalog_manager.cc:5484] T 95d1c7bd2c424900b399fac3fdb5c9ac P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:50.585820 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d1c7bd2c424900b399fac3fdb5c9ac with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:18:50.586784 20058 tablet_replica.cc:330] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:18:50.586990 20058 raft_consensus.cc:2229] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:50.587216 20058 raft_consensus.cc:2258] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:50.587756 20058 ts_tablet_manager.cc:1875] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:50.592037 20058 ts_tablet_manager.cc:1888] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:50.592149 20058 log.cc:1196] T 95d1c7bd2c424900b399fac3fdb5c9ac P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/95d1c7bd2c424900b399fac3fdb5c9ac
I20240502 08:18:50.592818 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 95d1c7bd2c424900b399fac3fdb5c9ac (table rebalancer_extra_table_10 [id=e77588602bd34d11a274be948cc9d52a]) successfully deleted
I20240502 08:18:50.999413 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:51.010764 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:51.051811 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:51.078023 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:51.102522 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:51.178975 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:51.186122 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:51.222730 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:51.424502 13120 consensus_queue.cc:237] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:51.426367 15486 raft_consensus.cc:1261] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:51.426524 12860 raft_consensus.cc:1261] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:51.427033 19793 consensus_queue.cc:1035] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:51.427534 19793 consensus_queue.cc:1035] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:51.434832 19793 raft_consensus.cc:2932] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:51.441201 12860 raft_consensus.cc:2932] T 91f185fbf27049a7992daa1d905e5575 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:51.443454 15486 raft_consensus.cc:2932] T 91f185fbf27049a7992daa1d905e5575 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:51.452458 12580 catalog_manager.cc:5484] T 91f185fbf27049a7992daa1d905e5575 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:51.479825 12708 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f185fbf27049a7992daa1d905e5575 with delete_type TABLET_DATA_TOMBSTONED (TS 29b035434087435cbaf4ebcac98eedee not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:51.489799 20147 tablet_replica.cc:330] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:51.492417 20147 raft_consensus.cc:2229] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:51.492647 20147 raft_consensus.cc:2258] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:51.493234 20147 ts_tablet_manager.cc:1875] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:51.499375 20147 ts_tablet_manager.cc:1888] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:18:51.499545 20147 log.cc:1196] T 91f185fbf27049a7992daa1d905e5575 P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/91f185fbf27049a7992daa1d905e5575
I20240502 08:18:51.500324 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 91f185fbf27049a7992daa1d905e5575 (table rebalance_test_table_13 [id=f2c6e6573e254b7d815efdcd481a81a2]) successfully deleted
I20240502 08:18:51.723495 12727 consensus_queue.cc:237] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:51.725666 12989 raft_consensus.cc:1261] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:51.726315 20097 consensus_queue.cc:1035] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:18:51.726406 12663 consensus_peers.cc:485] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee -> Peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Couldn't send request to peer 9333cb10227b4801b4fa7c24e22e131b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1581261b4c144a9b670786eb3d26466. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:51.727108 13120 raft_consensus.cc:1261] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:51.727728 20032 consensus_queue.cc:1035] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:51.730010 20032 raft_consensus.cc:2932] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:51.730633 13120 raft_consensus.cc:2932] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:51.738806 12989 raft_consensus.cc:2932] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:51.741458 12586 catalog_manager.cc:5484] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:51.799754 12989 consensus_queue.cc:237] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:51.802626 12860 raft_consensus.cc:1261] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:51.803279 12927 consensus_peers.cc:485] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 -> Peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Couldn't send request to peer 10f0e36203914b85b77e78609ffced98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fa42a21c83942f789dcebed741b52ae. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:51.803823 18405 consensus_queue.cc:1035] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:51.802393 13120 raft_consensus.cc:1261] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:51.807996 18405 consensus_queue.cc:1035] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:51.810451 20274 raft_consensus.cc:2932] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:51.811040 13119 raft_consensus.cc:2932] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:51.815007 12860 raft_consensus.cc:2932] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:51.819533 12585 catalog_manager.cc:5484] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:51.859202 12860 consensus_queue.cc:237] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:51.862854 15622 raft_consensus.cc:1261] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:51.863276 12989 raft_consensus.cc:1261] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:51.863423 12796 consensus_peers.cc:485] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a -> Peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Couldn't send request to peer db7893bb29a047b3b82cc60bfd8930a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e3cc08d9534400ab7ee8db08e6e7ee0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:51.863772 20267 consensus_queue.cc:1035] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:51.864077 19947 consensus_queue.cc:1035] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:51.868397 19947 raft_consensus.cc:2932] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:51.874537 12989 raft_consensus.cc:2932] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:51.874706 15620 raft_consensus.cc:2932] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:51.877418 12580 catalog_manager.cc:5484] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index -1 to 2, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:18:51.910488 15754 consensus_queue.cc:237] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:51.912583 12989 raft_consensus.cc:1261] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:51.912634 15887 raft_consensus.cc:1261] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:51.913057 20314 consensus_queue.cc:1035] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:51.913304 20112 consensus_queue.cc:1035] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:51.913772 15688 consensus_peers.cc:485] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Couldn't send request to peer 29b035434087435cbaf4ebcac98eedee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b73852b46f7e4db89e30d83edafa6413. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:51.916837 20112 raft_consensus.cc:2932] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:51.917683 12991 raft_consensus.cc:2932] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:51.920596 15887 raft_consensus.cc:2932] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:51.922730 12586 catalog_manager.cc:5484] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:51.955385 12727 consensus_queue.cc:237] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:51.957644 12991 raft_consensus.cc:1261] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:51.957952 13119 raft_consensus.cc:1261] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:51.957965 12663 consensus_peers.cc:485] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee -> Peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Couldn't send request to peer 9333cb10227b4801b4fa7c24e22e131b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b87724cd8beb4449b272f972c85a6666. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:51.958441 20097 consensus_queue.cc:1035] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:51.958978 20097 consensus_queue.cc:1035] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:51.961866 20097 raft_consensus.cc:2932] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:51.962386 13119 raft_consensus.cc:2932] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:51.967758 12580 catalog_manager.cc:5484] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:51.968987 12991 raft_consensus.cc:2932] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:51.998567 12991 consensus_queue.cc:237] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:52.000630 13119 raft_consensus.cc:1261] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:52.000877 12927 consensus_peers.cc:485] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 -> Peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Couldn't send request to peer 10f0e36203914b85b77e78609ffced98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3583c839d364bae819928eca86010ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:52.001579 20277 consensus_queue.cc:1035] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:52.001721 12862 raft_consensus.cc:1261] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:52.002247 20277 consensus_queue.cc:1035] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:52.004712 20274 raft_consensus.cc:2932] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:52.005285 12862 raft_consensus.cc:2932] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:52.008689 13119 raft_consensus.cc:2932] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:52.011830 12585 catalog_manager.cc:5484] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:18:52.065789 15887 consensus_queue.cc:237] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:52.068475 15487 raft_consensus.cc:1261] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:52.068502 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c60b24cf2d4740ae9b8829e288bc2530. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:52.068571 15754 raft_consensus.cc:1261] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:52.069247 18901 consensus_queue.cc:1035] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:52.069867 20209 consensus_queue.cc:1035] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:52.073069 18901 raft_consensus.cc:2932] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:52.075820 15754 raft_consensus.cc:2932] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:52.079399 15487 raft_consensus.cc:2932] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:52.080549 12580 catalog_manager.cc:5484] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:52.117476 13119 consensus_queue.cc:237] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
W20240502 08:18:52.120028 13056 consensus_peers.cc:485] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): Couldn't send request to peer 60b6e265e7754b7187307d05ef405dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 109bc82d7ff94f059dcbc80555b88b1b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:52.120096 15885 raft_consensus.cc:1261] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:52.120589 12727 raft_consensus.cc:1261] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:52.120671 20387 consensus_queue.cc:1035] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:52.121102 19793 consensus_queue.cc:1035] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:52.123703 19793 raft_consensus.cc:2932] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:52.127218 15887 raft_consensus.cc:2932] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:52.127312 12727 raft_consensus.cc:2932] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:52.132496 12580 catalog_manager.cc:5484] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:52.149402 20399 ts_tablet_manager.cc:902] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:52.157821 20399 tablet_copy_client.cc:308] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:18:52.159222 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet f1581261b4c144a9b670786eb3d26466 from peer 9333cb10227b4801b4fa7c24e22e131b ({username='slave'} at 127.6.5.7:35555)
I20240502 08:18:52.159348 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet f1581261b4c144a9b670786eb3d26466 from peer 9333cb10227b4801b4fa7c24e22e131b at {username='slave'} at 127.6.5.7:35555: session id = 9333cb10227b4801b4fa7c24e22e131b-f1581261b4c144a9b670786eb3d26466
I20240502 08:18:52.162263 12751 tablet_copy_source_session.cc:205] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:52.163617 20399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1581261b4c144a9b670786eb3d26466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.169868 20399 tablet_copy_client.cc:783] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:52.170076 20399 tablet_copy_client.cc:647] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:52.172413 20399 tablet_copy_client.cc:523] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:52.173884 12862 consensus_queue.cc:237] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:52.175218 20399 tablet_bootstrap.cc:492] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:52.177584 13119 raft_consensus.cc:1261] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:52.177584 12991 raft_consensus.cc:1261] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:52.178165 20267 consensus_queue.cc:1035] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:52.178598 19947 consensus_queue.cc:1035] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:18:52.179198 12795 consensus_peers.cc:485] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a -> Peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Couldn't send request to peer 9333cb10227b4801b4fa7c24e22e131b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5be94bf194b04e1d965274a968f30fbf. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:52.182014 20267 raft_consensus.cc:2932] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:52.182685 13119 raft_consensus.cc:2932] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:52.189623 12585 catalog_manager.cc:5484] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:52.194764 12991 raft_consensus.cc:2932] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:52.226140 20399 tablet_bootstrap.cc:492] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:52.226610 20399 tablet_bootstrap.cc:492] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap complete.
I20240502 08:18:52.226862 20399 ts_tablet_manager.cc:1367] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.052s user 0.043s sys 0.004s
I20240502 08:18:52.227644 20399 raft_consensus.cc:348] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:52.227880 20399 raft_consensus.cc:729] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.228121 20399 consensus_queue.cc:260] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:52.228806 20399 ts_tablet_manager.cc:1398] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:18:52.244823 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 9333cb10227b4801b4fa7c24e22e131b-f1581261b4c144a9b670786eb3d26466 received from {username='slave'} at 127.6.5.7:35555
I20240502 08:18:52.244948 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 9333cb10227b4801b4fa7c24e22e131b-f1581261b4c144a9b670786eb3d26466 on tablet f1581261b4c144a9b670786eb3d26466 with peer 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:52.283898 13119 consensus_queue.cc:237] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:52.286284 12727 raft_consensus.cc:1261] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:52.286545 12862 raft_consensus.cc:1261] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:52.286737 19793 consensus_queue.cc:1035] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:18:52.287024 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a8e5440619a4cb0b40dd286862184cd. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:52.287094 20387 consensus_queue.cc:1035] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:52.289402 19793 raft_consensus.cc:2932] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:52.289988 12862 raft_consensus.cc:2932] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:52.293036 12727 raft_consensus.cc:2932] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:52.295270 12585 catalog_manager.cc:5484] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:52.317580 20442 ts_tablet_manager.cc:902] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee: Initiating tablet copy from peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:52.320412 20442 tablet_copy_client.cc:308] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.7:38851
I20240502 08:18:52.321242 15775 tablet_copy_service.cc:140] P 9333cb10227b4801b4fa7c24e22e131b: Received BeginTabletCopySession request for tablet b73852b46f7e4db89e30d83edafa6413 from peer 29b035434087435cbaf4ebcac98eedee ({username='slave'} at 127.6.5.1:37709)
I20240502 08:18:52.321410 15775 tablet_copy_service.cc:161] P 9333cb10227b4801b4fa7c24e22e131b: Beginning new tablet copy session on tablet b73852b46f7e4db89e30d83edafa6413 from peer 29b035434087435cbaf4ebcac98eedee at {username='slave'} at 127.6.5.1:37709: session id = 29b035434087435cbaf4ebcac98eedee-b73852b46f7e4db89e30d83edafa6413
I20240502 08:18:52.323752 15775 tablet_copy_source_session.cc:205] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:52.325685 20442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b73852b46f7e4db89e30d83edafa6413. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.333290 20442 tablet_copy_client.cc:783] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:52.333534 20442 tablet_copy_client.cc:647] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:52.335685 20442 tablet_copy_client.cc:523] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:52.338011 20442 tablet_bootstrap.cc:492] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:52.344403 12727 consensus_queue.cc:237] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:52.344866 20442 tablet_bootstrap.cc:492] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:52.345288 20442 tablet_bootstrap.cc:492] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee: Bootstrap complete.
I20240502 08:18:52.345544 20442 ts_tablet_manager.cc:1367] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 08:18:52.346390 20442 raft_consensus.cc:348] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:52.346766 20442 raft_consensus.cc:729] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.347026 20442 consensus_queue.cc:260] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:52.348953 20442 ts_tablet_manager.cc:1398] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:52.349704 15775 tablet_copy_service.cc:342] P 9333cb10227b4801b4fa7c24e22e131b: Request end of tablet copy session 29b035434087435cbaf4ebcac98eedee-b73852b46f7e4db89e30d83edafa6413 received from {username='slave'} at 127.6.5.1:37709
I20240502 08:18:52.349813 15775 tablet_copy_service.cc:433] P 9333cb10227b4801b4fa7c24e22e131b: ending tablet copy session 29b035434087435cbaf4ebcac98eedee-b73852b46f7e4db89e30d83edafa6413 on tablet b73852b46f7e4db89e30d83edafa6413 with peer 29b035434087435cbaf4ebcac98eedee
I20240502 08:18:52.350047 15753 raft_consensus.cc:1261] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:52.350445 12664 consensus_peers.cc:485] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee -> Peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): Couldn't send request to peer 60b6e265e7754b7187307d05ef405dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a542eb208b254055b02765aceba5b405. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:52.350602 20097 consensus_queue.cc:1035] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:52.351029 15887 raft_consensus.cc:1261] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:52.352419 20097 consensus_queue.cc:1035] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:52.368180 20097 raft_consensus.cc:2932] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:52.368649 15887 raft_consensus.cc:2932] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:52.371757 15753 raft_consensus.cc:2932] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:52.380941 12580 catalog_manager.cc:5484] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:52.414932 15487 consensus_queue.cc:237] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:52.417805 12860 raft_consensus.cc:1261] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:52.417903 13120 raft_consensus.cc:1261] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:52.418376 19614 consensus_queue.cc:1035] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:52.419674 19614 consensus_queue.cc:1035] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:52.422232 15419 consensus_peers.cc:485] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Couldn't send request to peer 29b035434087435cbaf4ebcac98eedee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b930dcb3863402296a205fdb65d6fc3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:52.423473 19614 raft_consensus.cc:2932] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:52.427476 20478 ts_tablet_manager.cc:902] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:52.429554 12861 raft_consensus.cc:2932] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:52.436802 12586 catalog_manager.cc:5484] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:52.442049 13120 raft_consensus.cc:2932] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:18:52.458854 20478 tablet_copy_client.cc:308] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:18:52.460014 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet 0fa42a21c83942f789dcebed741b52ae from peer 10f0e36203914b85b77e78609ffced98 ({username='slave'} at 127.6.5.8:33311)
I20240502 08:18:52.460239 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet 0fa42a21c83942f789dcebed741b52ae from peer 10f0e36203914b85b77e78609ffced98 at {username='slave'} at 127.6.5.8:33311: session id = 10f0e36203914b85b77e78609ffced98-0fa42a21c83942f789dcebed741b52ae
I20240502 08:18:52.469072 13012 tablet_copy_source_session.cc:205] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:52.472296 20399 ts_tablet_manager.cc:902] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:52.475713 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa42a21c83942f789dcebed741b52ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.489593 20478 tablet_copy_client.cc:783] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:52.489822 20478 tablet_copy_client.cc:647] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:52.490257 20483 ts_tablet_manager.cc:902] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3: Initiating tablet copy from peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:52.494084 20483 tablet_copy_client.cc:308] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:36933
I20240502 08:18:52.494848 12882 tablet_copy_service.cc:140] P e1dd14f7ca764a1490df419f13f2804a: Received BeginTabletCopySession request for tablet 2e3cc08d9534400ab7ee8db08e6e7ee0 from peer db7893bb29a047b3b82cc60bfd8930a3 ({username='slave'} at 127.6.5.5:45483)
I20240502 08:18:52.495060 12882 tablet_copy_service.cc:161] P e1dd14f7ca764a1490df419f13f2804a: Beginning new tablet copy session on tablet 2e3cc08d9534400ab7ee8db08e6e7ee0 from peer db7893bb29a047b3b82cc60bfd8930a3 at {username='slave'} at 127.6.5.5:45483: session id = db7893bb29a047b3b82cc60bfd8930a3-2e3cc08d9534400ab7ee8db08e6e7ee0
I20240502 08:18:52.500494 12882 tablet_copy_source_session.cc:205] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:52.503060 20483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e3cc08d9534400ab7ee8db08e6e7ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.503763 20488 ts_tablet_manager.cc:902] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:52.510118 20478 tablet_copy_client.cc:523] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:52.515966 20399 tablet_copy_client.cc:308] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:18:52.516605 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet b87724cd8beb4449b272f972c85a6666 from peer 9333cb10227b4801b4fa7c24e22e131b ({username='slave'} at 127.6.5.7:35555)
I20240502 08:18:52.516739 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet b87724cd8beb4449b272f972c85a6666 from peer 9333cb10227b4801b4fa7c24e22e131b at {username='slave'} at 127.6.5.7:35555: session id = 9333cb10227b4801b4fa7c24e22e131b-b87724cd8beb4449b272f972c85a6666
I20240502 08:18:52.518420 20478 tablet_bootstrap.cc:492] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:52.519577 12751 tablet_copy_source_session.cc:205] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:52.520973 20399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b87724cd8beb4449b272f972c85a6666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.531175 20399 tablet_copy_client.cc:783] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:52.531526 20399 tablet_copy_client.cc:647] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:52.533149 20399 tablet_copy_client.cc:523] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:52.535712 20483 tablet_copy_client.cc:783] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:52.536693 20483 tablet_copy_client.cc:647] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:52.537256 20399 tablet_bootstrap.cc:492] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:52.544950 20483 tablet_copy_client.cc:523] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:52.548887 15753 raft_consensus.cc:1203] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:52.561038 20488 tablet_copy_client.cc:308] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:18:52.561200 20483 tablet_bootstrap.cc:492] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:18:52.561635 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet a3583c839d364bae819928eca86010ec from peer 10f0e36203914b85b77e78609ffced98 ({username='slave'} at 127.6.5.8:33311)
I20240502 08:18:52.561831 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet a3583c839d364bae819928eca86010ec from peer 10f0e36203914b85b77e78609ffced98 at {username='slave'} at 127.6.5.8:33311: session id = 10f0e36203914b85b77e78609ffced98-a3583c839d364bae819928eca86010ec
I20240502 08:18:52.564682 13012 tablet_copy_source_session.cc:205] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:52.566033 20488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3583c839d364bae819928eca86010ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.568300 20505 ts_tablet_manager.cc:902] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:52.579963 20488 tablet_copy_client.cc:783] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:52.580180 20488 tablet_copy_client.cc:647] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:52.584041 20488 tablet_copy_client.cc:523] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:52.592006 20488 tablet_bootstrap.cc:492] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:18:52.593744 20483 tablet_bootstrap.cc:492] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:52.594189 20483 tablet_bootstrap.cc:492] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap complete.
I20240502 08:18:52.594517 20483 ts_tablet_manager.cc:1367] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.006s
I20240502 08:18:52.595350 20483 raft_consensus.cc:348] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:52.595806 20505 tablet_copy_client.cc:308] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:18:52.596086 20483 raft_consensus.cc:729] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.596436 20483 consensus_queue.cc:260] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:18:52.596907 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet 109bc82d7ff94f059dcbc80555b88b1b from peer 60b6e265e7754b7187307d05ef405dd4 ({username='slave'} at 127.6.5.3:46755)
I20240502 08:18:52.597050 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet 109bc82d7ff94f059dcbc80555b88b1b from peer 60b6e265e7754b7187307d05ef405dd4 at {username='slave'} at 127.6.5.3:46755: session id = 60b6e265e7754b7187307d05ef405dd4-109bc82d7ff94f059dcbc80555b88b1b
I20240502 08:18:52.597179 20483 ts_tablet_manager.cc:1398] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:18:52.599467 13142 tablet_copy_source_session.cc:205] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:52.600539 12882 tablet_copy_service.cc:342] P e1dd14f7ca764a1490df419f13f2804a: Request end of tablet copy session db7893bb29a047b3b82cc60bfd8930a3-2e3cc08d9534400ab7ee8db08e6e7ee0 received from {username='slave'} at 127.6.5.5:45483
I20240502 08:18:52.600682 12882 tablet_copy_service.cc:433] P e1dd14f7ca764a1490df419f13f2804a: ending tablet copy session db7893bb29a047b3b82cc60bfd8930a3-2e3cc08d9534400ab7ee8db08e6e7ee0 on tablet 2e3cc08d9534400ab7ee8db08e6e7ee0 with peer db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:18:52.612234 20515 ts_tablet_manager.cc:902] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:52.616211 20505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109bc82d7ff94f059dcbc80555b88b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.624275 20478 tablet_bootstrap.cc:492] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:52.627467 20478 tablet_bootstrap.cc:492] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98: Bootstrap complete.
I20240502 08:18:52.627707 20478 ts_tablet_manager.cc:1367] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.109s user 0.061s sys 0.010s
I20240502 08:18:52.628322 20478 raft_consensus.cc:348] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:52.628532 20478 raft_consensus.cc:729] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.628729 20478 consensus_queue.cc:260] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:52.631107 20478 ts_tablet_manager.cc:1398] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:18:52.632030 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session 10f0e36203914b85b77e78609ffced98-0fa42a21c83942f789dcebed741b52ae received from {username='slave'} at 127.6.5.8:33311
I20240502 08:18:52.632169 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session 10f0e36203914b85b77e78609ffced98-0fa42a21c83942f789dcebed741b52ae on tablet 0fa42a21c83942f789dcebed741b52ae with peer 10f0e36203914b85b77e78609ffced98
I20240502 08:18:52.658502 20505 tablet_copy_client.cc:783] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:52.658748 20505 tablet_copy_client.cc:647] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:52.660631 20505 tablet_copy_client.cc:523] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:52.671437 20505 tablet_bootstrap.cc:492] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:52.675318 20399 tablet_bootstrap.cc:492] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:52.675701 20399 tablet_bootstrap.cc:492] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap complete.
I20240502 08:18:52.675940 20399 ts_tablet_manager.cc:1367] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.139s user 0.059s sys 0.008s
I20240502 08:18:52.676597 20399 raft_consensus.cc:348] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:52.676857 20399 raft_consensus.cc:729] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.677088 20399 consensus_queue.cc:260] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:52.677763 20399 ts_tablet_manager.cc:1398] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:18:52.678453 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 9333cb10227b4801b4fa7c24e22e131b-b87724cd8beb4449b272f972c85a6666 received from {username='slave'} at 127.6.5.7:35555
I20240502 08:18:52.678586 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 9333cb10227b4801b4fa7c24e22e131b-b87724cd8beb4449b272f972c85a6666 on tablet b87724cd8beb4449b272f972c85a6666 with peer 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:52.699932 20488 tablet_bootstrap.cc:492] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:52.700424 20488 tablet_bootstrap.cc:492] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98: Bootstrap complete.
I20240502 08:18:52.700795 20488 ts_tablet_manager.cc:1367] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.109s user 0.046s sys 0.009s
I20240502 08:18:52.701529 20488 raft_consensus.cc:348] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:52.701890 20488 raft_consensus.cc:729] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.702169 20488 consensus_queue.cc:260] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:18:52.702908 20488 ts_tablet_manager.cc:1398] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:52.706307 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session 10f0e36203914b85b77e78609ffced98-a3583c839d364bae819928eca86010ec received from {username='slave'} at 127.6.5.8:33311
I20240502 08:18:52.706442 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session 10f0e36203914b85b77e78609ffced98-a3583c839d364bae819928eca86010ec on tablet a3583c839d364bae819928eca86010ec with peer 10f0e36203914b85b77e78609ffced98
I20240502 08:18:52.712625 20522 ts_tablet_manager.cc:902] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:18:52.717212 20515 tablet_copy_client.cc:308] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.8:46303
I20240502 08:18:52.717952 15907 tablet_copy_service.cc:140] P 10f0e36203914b85b77e78609ffced98: Received BeginTabletCopySession request for tablet c60b24cf2d4740ae9b8829e288bc2530 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:41995)
I20240502 08:18:52.718071 15907 tablet_copy_service.cc:161] P 10f0e36203914b85b77e78609ffced98: Beginning new tablet copy session on tablet c60b24cf2d4740ae9b8829e288bc2530 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:41995: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-c60b24cf2d4740ae9b8829e288bc2530
I20240502 08:18:52.720500 15907 tablet_copy_source_session.cc:205] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:52.721979 20515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60b24cf2d4740ae9b8829e288bc2530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.724035 20505 tablet_bootstrap.cc:492] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:52.724444 20505 tablet_bootstrap.cc:492] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4: Bootstrap complete.
I20240502 08:18:52.724700 20505 ts_tablet_manager.cc:1367] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.002s
I20240502 08:18:52.725423 20505 raft_consensus.cc:348] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:52.725639 20505 raft_consensus.cc:729] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.726205 20515 tablet_copy_client.cc:783] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:52.726397 20515 tablet_copy_client.cc:647] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:52.727937 20515 tablet_copy_client.cc:523] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:52.729115 20522 tablet_copy_client.cc:308] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:18:52.729511 20505 consensus_queue.cc:260] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:52.729987 20505 ts_tablet_manager.cc:1398] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:18:52.730311 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet 7a8e5440619a4cb0b40dd286862184cd from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:37199)
I20240502 08:18:52.730496 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet 7a8e5440619a4cb0b40dd286862184cd from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:37199: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-7a8e5440619a4cb0b40dd286862184cd
I20240502 08:18:52.730754 13141 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session 60b6e265e7754b7187307d05ef405dd4-109bc82d7ff94f059dcbc80555b88b1b received from {username='slave'} at 127.6.5.3:46755
I20240502 08:18:52.730881 13141 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session 60b6e265e7754b7187307d05ef405dd4-109bc82d7ff94f059dcbc80555b88b1b on tablet 109bc82d7ff94f059dcbc80555b88b1b with peer 60b6e265e7754b7187307d05ef405dd4
I20240502 08:18:52.732728 20515 tablet_bootstrap.cc:492] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:52.733088 13142 tablet_copy_source_session.cc:205] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:52.734089 20522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a8e5440619a4cb0b40dd286862184cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.743739 15887 consensus_queue.cc:237] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:52.745863 15622 raft_consensus.cc:1261] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:52.746241 15753 raft_consensus.cc:1261] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:52.746546 19735 consensus_queue.cc:1035] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:18:52.747215 15818 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Couldn't send request to peer 6a0b05f3250d415590b0edf6b44fbe1f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e882f58c7f344f7929b8a30bd65fa44. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:52.747401 18901 consensus_queue.cc:1035] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:52.752046 20505 ts_tablet_manager.cc:902] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:52.752530 20522 tablet_copy_client.cc:783] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:52.752734 20522 tablet_copy_client.cc:647] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:52.756191 20522 tablet_copy_client.cc:523] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:52.768178 20515 tablet_bootstrap.cc:492] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:52.768564 20515 tablet_bootstrap.cc:492] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:18:52.768798 20515 ts_tablet_manager.cc:1367] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.004s
I20240502 08:18:52.769748 20522 tablet_bootstrap.cc:492] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:52.770798 20515 raft_consensus.cc:348] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:52.770944 20515 raft_consensus.cc:729] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.771257 20515 consensus_queue.cc:260] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:52.771770 20515 ts_tablet_manager.cc:1398] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240502 08:18:52.773564 15907 tablet_copy_service.cc:342] P 10f0e36203914b85b77e78609ffced98: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-c60b24cf2d4740ae9b8829e288bc2530 received from {username='slave'} at 127.6.5.6:41995
I20240502 08:18:52.773669 15907 tablet_copy_service.cc:433] P 10f0e36203914b85b77e78609ffced98: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-c60b24cf2d4740ae9b8829e288bc2530 on tablet c60b24cf2d4740ae9b8829e288bc2530 with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:52.773607 18901 raft_consensus.cc:2932] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:52.786128 15622 raft_consensus.cc:2932] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:52.788209 20399 ts_tablet_manager.cc:902] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b: Initiating tablet copy from peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:52.796033 12730 raft_consensus.cc:1203] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:52.805894 12580 catalog_manager.cc:5484] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:52.809010 20442 ts_tablet_manager.cc:902] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:52.820502 15753 raft_consensus.cc:2932] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:18:52.824450 15887 raft_consensus.cc:1203] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:52.827590 15487 raft_consensus.cc:1203] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:52.865068 20399 tablet_copy_client.cc:308] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:36933
I20240502 08:18:52.865710 12882 tablet_copy_service.cc:140] P e1dd14f7ca764a1490df419f13f2804a: Received BeginTabletCopySession request for tablet 5be94bf194b04e1d965274a968f30fbf from peer 9333cb10227b4801b4fa7c24e22e131b ({username='slave'} at 127.6.5.7:57025)
I20240502 08:18:52.865856 12882 tablet_copy_service.cc:161] P e1dd14f7ca764a1490df419f13f2804a: Beginning new tablet copy session on tablet 5be94bf194b04e1d965274a968f30fbf from peer 9333cb10227b4801b4fa7c24e22e131b at {username='slave'} at 127.6.5.7:57025: session id = 9333cb10227b4801b4fa7c24e22e131b-5be94bf194b04e1d965274a968f30fbf
I20240502 08:18:52.866487 20505 tablet_copy_client.cc:308] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:18:52.867218 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet a542eb208b254055b02765aceba5b405 from peer 60b6e265e7754b7187307d05ef405dd4 ({username='slave'} at 127.6.5.3:52517)
I20240502 08:18:52.867362 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet a542eb208b254055b02765aceba5b405 from peer 60b6e265e7754b7187307d05ef405dd4 at {username='slave'} at 127.6.5.3:52517: session id = 60b6e265e7754b7187307d05ef405dd4-a542eb208b254055b02765aceba5b405
I20240502 08:18:52.868451 12882 tablet_copy_source_session.cc:205] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:52.870110 12751 tablet_copy_source_session.cc:205] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:52.871292 20505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a542eb208b254055b02765aceba5b405. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.873267 20399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be94bf194b04e1d965274a968f30fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.877944 20505 tablet_copy_client.cc:783] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:52.878190 20505 tablet_copy_client.cc:647] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:52.878178 20442 tablet_copy_client.cc:308] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:18:52.879093 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet 3b930dcb3863402296a205fdb65d6fc3 from peer 29b035434087435cbaf4ebcac98eedee ({username='slave'} at 127.6.5.1:33495)
I20240502 08:18:52.879238 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet 3b930dcb3863402296a205fdb65d6fc3 from peer 29b035434087435cbaf4ebcac98eedee at {username='slave'} at 127.6.5.1:33495: session id = 29b035434087435cbaf4ebcac98eedee-3b930dcb3863402296a205fdb65d6fc3
I20240502 08:18:52.879683 20505 tablet_copy_client.cc:523] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:52.881232 20399 tablet_copy_client.cc:783] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:52.881443 20399 tablet_copy_client.cc:647] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:52.881879 15508 tablet_copy_source_session.cc:205] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:52.882817 20399 tablet_copy_client.cc:523] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:52.884264 20442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b930dcb3863402296a205fdb65d6fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.886031 20505 tablet_bootstrap.cc:492] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:52.895150 20442 tablet_copy_client.cc:783] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:52.895342 20442 tablet_copy_client.cc:647] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:52.898293 20505 tablet_bootstrap.cc:492] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:52.898663 20505 tablet_bootstrap.cc:492] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4: Bootstrap complete.
I20240502 08:18:52.898900 20505 ts_tablet_manager.cc:1367] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240502 08:18:52.899616 20505 raft_consensus.cc:348] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:52.899829 20505 raft_consensus.cc:729] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.900089 20505 consensus_queue.cc:260] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:52.900393 20442 tablet_copy_client.cc:523] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:52.901613 20399 tablet_bootstrap.cc:492] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:52.906551 20505 ts_tablet_manager.cc:1398] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 08:18:52.908962 20442 tablet_bootstrap.cc:492] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:18:52.908946 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 60b6e265e7754b7187307d05ef405dd4-a542eb208b254055b02765aceba5b405 received from {username='slave'} at 127.6.5.3:52517
I20240502 08:18:52.909121 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 60b6e265e7754b7187307d05ef405dd4-a542eb208b254055b02765aceba5b405 on tablet a542eb208b254055b02765aceba5b405 with peer 60b6e265e7754b7187307d05ef405dd4
I20240502 08:18:52.927704 20442 tablet_bootstrap.cc:492] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:52.928148 20442 tablet_bootstrap.cc:492] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee: Bootstrap complete.
I20240502 08:18:52.928462 20442 ts_tablet_manager.cc:1367] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.001s
I20240502 08:18:52.929875 20442 raft_consensus.cc:348] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:52.930200 20442 raft_consensus.cc:729] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.930516 20442 consensus_queue.cc:260] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:52.932639 20442 ts_tablet_manager.cc:1398] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:52.933332 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session 29b035434087435cbaf4ebcac98eedee-3b930dcb3863402296a205fdb65d6fc3 received from {username='slave'} at 127.6.5.1:33495
I20240502 08:18:52.933455 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session 29b035434087435cbaf4ebcac98eedee-3b930dcb3863402296a205fdb65d6fc3 on tablet 3b930dcb3863402296a205fdb65d6fc3 with peer 29b035434087435cbaf4ebcac98eedee
I20240502 08:18:52.935454 12991 consensus_queue.cc:237] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:52.938184 13119 raft_consensus.cc:1261] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:52.938375 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83e0c78cb2374ba483315ca8a827e532. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:52.938688 12861 raft_consensus.cc:1261] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:52.939213 20274 consensus_queue.cc:1035] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:52.939625 20277 consensus_queue.cc:1035] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:52.956408 20274 raft_consensus.cc:2932] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:52.957068 13119 raft_consensus.cc:2932] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:52.961290 15753 raft_consensus.cc:1203] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:52.966020 12861 raft_consensus.cc:2932] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:52.968245 20399 tablet_bootstrap.cc:492] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:52.968323 12585 catalog_manager.cc:5484] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:52.968631 20399 tablet_bootstrap.cc:492] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap complete.
I20240502 08:18:52.968886 20399 ts_tablet_manager.cc:1367] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.067s user 0.040s sys 0.008s
I20240502 08:18:52.969595 20399 raft_consensus.cc:348] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:52.969820 20399 raft_consensus.cc:729] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.970037 20399 consensus_queue.cc:260] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:52.970620 20399 ts_tablet_manager.cc:1398] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 08:18:52.972187 12882 tablet_copy_service.cc:342] P e1dd14f7ca764a1490df419f13f2804a: Request end of tablet copy session 9333cb10227b4801b4fa7c24e22e131b-5be94bf194b04e1d965274a968f30fbf received from {username='slave'} at 127.6.5.7:57025
I20240502 08:18:52.972293 12882 tablet_copy_service.cc:433] P e1dd14f7ca764a1490df419f13f2804a: ending tablet copy session 9333cb10227b4801b4fa7c24e22e131b-5be94bf194b04e1d965274a968f30fbf on tablet 5be94bf194b04e1d965274a968f30fbf with peer 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:52.981954 20522 tablet_bootstrap.cc:492] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:52.982285 20522 tablet_bootstrap.cc:492] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:18:52.982509 20522 ts_tablet_manager.cc:1367] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.213s user 0.043s sys 0.010s
I20240502 08:18:52.983069 20522 raft_consensus.cc:348] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:52.983237 20522 raft_consensus.cc:729] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.983402 20522 consensus_queue.cc:260] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:52.983850 20522 ts_tablet_manager.cc:1398] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:18:52.984431 13142 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-7a8e5440619a4cb0b40dd286862184cd received from {username='slave'} at 127.6.5.6:37199
I20240502 08:18:52.984514 13142 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-7a8e5440619a4cb0b40dd286862184cd on tablet 7a8e5440619a4cb0b40dd286862184cd with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:52.990410 15620 raft_consensus.cc:1203] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:52.996687 12991 raft_consensus.cc:1203] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:53.045668 15753 consensus_queue.cc:237] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
W20240502 08:18:53.060102 15688 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): Couldn't send request to peer 60b6e265e7754b7187307d05ef405dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18c58e2a9a814bcb8ff6bb9e3011084f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:53.060235 15620 raft_consensus.cc:1261] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:53.061883 20112 consensus_queue.cc:1035] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:53.065184 15887 raft_consensus.cc:1203] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:53.067390 15885 raft_consensus.cc:1261] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:53.067870 20590 consensus_queue.cc:1035] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:53.075040 20112 raft_consensus.cc:2932] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:53.075659 15885 raft_consensus.cc:2932] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:53.079427 15620 raft_consensus.cc:2932] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:53.086477 12586 catalog_manager.cc:5484] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:53.120417 15486 consensus_queue.cc:237] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
W20240502 08:18:53.122601 15418 consensus_peers.cc:485] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Couldn't send request to peer 9333cb10227b4801b4fa7c24e22e131b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c65f66921e73457eb07acec0670b2426. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:53.122695 12861 raft_consensus.cc:1261] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:53.122718 12991 raft_consensus.cc:1261] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:53.123209 20475 consensus_queue.cc:1035] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:53.123477 19614 consensus_queue.cc:1035] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:53.125952 19614 raft_consensus.cc:2932] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:53.126552 12991 raft_consensus.cc:2932] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:53.129854 12861 raft_consensus.cc:2932] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:53.131637 12585 catalog_manager.cc:5484] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:53.168983 15753 consensus_queue.cc:237] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:53.170892 15620 raft_consensus.cc:1261] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:53.171058 12730 raft_consensus.cc:1261] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:18:53.171465 15688 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a9d9100dbaa4861ba1d2a1455f9756d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:53.171490 20112 consensus_queue.cc:1035] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:53.171769 20590 consensus_queue.cc:1035] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:53.174908 20590 raft_consensus.cc:2932] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:53.176632 12991 raft_consensus.cc:1203] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:53.180564 12730 raft_consensus.cc:2932] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:53.181365 15620 raft_consensus.cc:2932] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:53.185948 12583 catalog_manager.cc:5484] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:53.223773 20638 ts_tablet_manager.cc:902] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f: Initiating tablet copy from peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:18:53.225401 20638 tablet_copy_client.cc:308] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.8:46303
I20240502 08:18:53.225972 15907 tablet_copy_service.cc:140] P 10f0e36203914b85b77e78609ffced98: Received BeginTabletCopySession request for tablet 0e882f58c7f344f7929b8a30bd65fa44 from peer 6a0b05f3250d415590b0edf6b44fbe1f ({username='slave'} at 127.6.5.4:32771)
I20240502 08:18:53.226099 15907 tablet_copy_service.cc:161] P 10f0e36203914b85b77e78609ffced98: Beginning new tablet copy session on tablet 0e882f58c7f344f7929b8a30bd65fa44 from peer 6a0b05f3250d415590b0edf6b44fbe1f at {username='slave'} at 127.6.5.4:32771: session id = 6a0b05f3250d415590b0edf6b44fbe1f-0e882f58c7f344f7929b8a30bd65fa44
I20240502 08:18:53.228883 15907 tablet_copy_source_session.cc:205] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:53.230101 20638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e882f58c7f344f7929b8a30bd65fa44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:53.235000 12730 raft_consensus.cc:1203] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:53.235381 20638 tablet_copy_client.cc:783] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:53.236783 20638 tablet_copy_client.cc:647] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:53.239997 20638 tablet_copy_client.cc:523] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:53.244907 20638 tablet_bootstrap.cc:492] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:18:53.252269 20638 tablet_bootstrap.cc:492] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:53.252615 20638 tablet_bootstrap.cc:492] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap complete.
I20240502 08:18:53.252835 20638 ts_tablet_manager.cc:1367] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 08:18:53.253484 20638 raft_consensus.cc:348] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:53.253638 20638 raft_consensus.cc:729] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:18:53.253820 20638 consensus_queue.cc:260] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:18:53.254390 20638 ts_tablet_manager.cc:1398] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240502 08:18:53.255174 15907 tablet_copy_service.cc:342] P 10f0e36203914b85b77e78609ffced98: Request end of tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-0e882f58c7f344f7929b8a30bd65fa44 received from {username='slave'} at 127.6.5.4:32771
I20240502 08:18:53.255277 15907 tablet_copy_service.cc:433] P 10f0e36203914b85b77e78609ffced98: ending tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-0e882f58c7f344f7929b8a30bd65fa44 on tablet 0e882f58c7f344f7929b8a30bd65fa44 with peer 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:18:53.257664 12730 consensus_queue.cc:237] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:53.261634 13119 raft_consensus.cc:1261] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:53.261811 12861 raft_consensus.cc:1261] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:53.262192 20655 consensus_queue.cc:1035] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:53.262497 20456 consensus_queue.cc:1035] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:18:53.262995 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b29a9d8d26b4a388c89fb8e3a0fefe3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:53.265178 20456 raft_consensus.cc:2932] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:53.265653 13119 raft_consensus.cc:2932] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:53.269084 12861 raft_consensus.cc:2932] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:53.271306 12583 catalog_manager.cc:5484] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:53.297814 15620 raft_consensus.cc:1203] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:53.309916 15620 consensus_queue.cc:237] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
W20240502 08:18:53.312628 15554 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b -> Peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): Couldn't send request to peer 60b6e265e7754b7187307d05ef405dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7082603ad2d6450e8863cd53b1897b2e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:53.312623 13119 raft_consensus.cc:1261] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:53.312680 15885 raft_consensus.cc:1261] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:53.313109 20266 consensus_queue.cc:1035] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:53.313586 19585 consensus_queue.cc:1035] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:18:53.317175 19585 raft_consensus.cc:2932] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:53.320104 15887 raft_consensus.cc:2932] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:53.325177 12586 catalog_manager.cc:5484] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:53.333691 13119 raft_consensus.cc:2932] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:53.353446 15753 raft_consensus.cc:1203] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:53.377496 20522 ts_tablet_manager.cc:902] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:18:53.380532 20522 tablet_copy_client.cc:308] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:18:53.381134 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet 83e0c78cb2374ba483315ca8a827e532 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:58785)
I20240502 08:18:53.381297 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet 83e0c78cb2374ba483315ca8a827e532 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:58785: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-83e0c78cb2374ba483315ca8a827e532
I20240502 08:18:53.383390 13012 tablet_copy_source_session.cc:205] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:53.384505 20522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83e0c78cb2374ba483315ca8a827e532. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:53.388804 20522 tablet_copy_client.cc:783] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:53.388985 20522 tablet_copy_client.cc:647] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:53.391814 20522 tablet_copy_client.cc:523] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:53.394834 20522 tablet_bootstrap.cc:492] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:53.434995 20522 tablet_bootstrap.cc:492] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:53.435382 20522 tablet_bootstrap.cc:492] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:18:53.435640 20522 ts_tablet_manager.cc:1367] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.041s user 0.038s sys 0.000s
I20240502 08:18:53.437353 20522 raft_consensus.cc:348] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:53.437620 20522 raft_consensus.cc:729] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:53.437835 20522 consensus_queue.cc:260] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:53.444196 20522 ts_tablet_manager.cc:1398] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 08:18:53.447593 20701 ts_tablet_manager.cc:902] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4: Initiating tablet copy from peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:53.448734 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-83e0c78cb2374ba483315ca8a827e532 received from {username='slave'} at 127.6.5.6:58785
I20240502 08:18:53.448866 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-83e0c78cb2374ba483315ca8a827e532 on tablet 83e0c78cb2374ba483315ca8a827e532 with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:53.468045 20701 tablet_copy_client.cc:308] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.7:38851
I20240502 08:18:53.468724 15775 tablet_copy_service.cc:140] P 9333cb10227b4801b4fa7c24e22e131b: Received BeginTabletCopySession request for tablet 18c58e2a9a814bcb8ff6bb9e3011084f from peer 60b6e265e7754b7187307d05ef405dd4 ({username='slave'} at 127.6.5.3:53567)
I20240502 08:18:53.468892 15775 tablet_copy_service.cc:161] P 9333cb10227b4801b4fa7c24e22e131b: Beginning new tablet copy session on tablet 18c58e2a9a814bcb8ff6bb9e3011084f from peer 60b6e265e7754b7187307d05ef405dd4 at {username='slave'} at 127.6.5.3:53567: session id = 60b6e265e7754b7187307d05ef405dd4-18c58e2a9a814bcb8ff6bb9e3011084f
I20240502 08:18:53.471784 15775 tablet_copy_source_session.cc:205] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:53.473037 20701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18c58e2a9a814bcb8ff6bb9e3011084f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:53.479444 20701 tablet_copy_client.cc:783] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:53.479725 20701 tablet_copy_client.cc:647] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:53.481256 20701 tablet_copy_client.cc:523] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:53.491405 20701 tablet_bootstrap.cc:492] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:53.502365 20701 tablet_bootstrap.cc:492] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:53.502918 20701 tablet_bootstrap.cc:492] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4: Bootstrap complete.
I20240502 08:18:53.503245 20701 ts_tablet_manager.cc:1367] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240502 08:18:53.504036 20701 raft_consensus.cc:348] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:53.504357 20701 raft_consensus.cc:729] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:53.504669 20701 consensus_queue.cc:260] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:53.505400 20701 ts_tablet_manager.cc:1398] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:18:53.507510 15775 tablet_copy_service.cc:342] P 9333cb10227b4801b4fa7c24e22e131b: Request end of tablet copy session 60b6e265e7754b7187307d05ef405dd4-18c58e2a9a814bcb8ff6bb9e3011084f received from {username='slave'} at 127.6.5.3:53567
I20240502 08:18:53.507651 15775 tablet_copy_service.cc:433] P 9333cb10227b4801b4fa7c24e22e131b: ending tablet copy session 60b6e265e7754b7187307d05ef405dd4-18c58e2a9a814bcb8ff6bb9e3011084f on tablet 18c58e2a9a814bcb8ff6bb9e3011084f with peer 60b6e265e7754b7187307d05ef405dd4
I20240502 08:18:53.605911 20738 ts_tablet_manager.cc:902] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:53.641585 20738 tablet_copy_client.cc:308] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.7:38851
I20240502 08:18:53.644802 13120 raft_consensus.cc:1203] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:53.646840 15775 tablet_copy_service.cc:140] P 9333cb10227b4801b4fa7c24e22e131b: Received BeginTabletCopySession request for tablet 6a9d9100dbaa4861ba1d2a1455f9756d from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:37259)
I20240502 08:18:53.646986 15775 tablet_copy_service.cc:161] P 9333cb10227b4801b4fa7c24e22e131b: Beginning new tablet copy session on tablet 6a9d9100dbaa4861ba1d2a1455f9756d from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:37259: session id = e1dd14f7ca764a1490df419f13f2804a-6a9d9100dbaa4861ba1d2a1455f9756d
I20240502 08:18:53.649657 15775 tablet_copy_source_session.cc:205] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:53.653638 20738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9d9100dbaa4861ba1d2a1455f9756d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:53.664592 20738 tablet_copy_client.cc:783] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:53.664804 20738 tablet_copy_client.cc:647] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:53.670450 20738 tablet_copy_client.cc:523] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:53.674403 20738 tablet_bootstrap.cc:492] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:18:53.687131 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:53.718187 20738 tablet_bootstrap.cc:492] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:53.719717 20738 tablet_bootstrap.cc:492] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:18:53.720769 20738 ts_tablet_manager.cc:1367] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.046s user 0.015s sys 0.022s
I20240502 08:18:53.723834 20738 raft_consensus.cc:348] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:53.724136 20738 raft_consensus.cc:729] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:18:53.724426 20738 consensus_queue.cc:260] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:53.725071 20738 ts_tablet_manager.cc:1398] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:18:53.728013 15775 tablet_copy_service.cc:342] P 9333cb10227b4801b4fa7c24e22e131b: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-6a9d9100dbaa4861ba1d2a1455f9756d received from {username='slave'} at 127.6.5.2:37259
I20240502 08:18:53.728168 15775 tablet_copy_service.cc:433] P 9333cb10227b4801b4fa7c24e22e131b: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-6a9d9100dbaa4861ba1d2a1455f9756d on tablet 6a9d9100dbaa4861ba1d2a1455f9756d with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:18:53.746806 20751 ts_tablet_manager.cc:902] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:18:53.756785 20522 ts_tablet_manager.cc:902] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:18:53.771198 20701 ts_tablet_manager.cc:902] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4: Initiating tablet copy from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:18:53.782286 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:53.821003 20522 tablet_copy_client.cc:308] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:18:53.824223 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet 5b29a9d8d26b4a388c89fb8e3a0fefe3 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:55577)
I20240502 08:18:53.824358 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet 5b29a9d8d26b4a388c89fb8e3a0fefe3 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:55577: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-5b29a9d8d26b4a388c89fb8e3a0fefe3
I20240502 08:18:53.827648 12751 tablet_copy_source_session.cc:205] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:53.848666 20522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b29a9d8d26b4a388c89fb8e3a0fefe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:53.853641 12989 raft_consensus.cc:1203] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:53.893302 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:53.895792 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:53.896608 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:53.910326 20701 tablet_copy_client.cc:308] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.6:44983
I20240502 08:18:53.911068 15642 tablet_copy_service.cc:140] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Received BeginTabletCopySession request for tablet 7082603ad2d6450e8863cd53b1897b2e from peer 60b6e265e7754b7187307d05ef405dd4 ({username='slave'} at 127.6.5.3:49097)
I20240502 08:18:53.911201 15642 tablet_copy_service.cc:161] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Beginning new tablet copy session on tablet 7082603ad2d6450e8863cd53b1897b2e from peer 60b6e265e7754b7187307d05ef405dd4 at {username='slave'} at 127.6.5.3:49097: session id = 60b6e265e7754b7187307d05ef405dd4-7082603ad2d6450e8863cd53b1897b2e
I20240502 08:18:53.912015 20522 tablet_copy_client.cc:783] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:53.912230 20522 tablet_copy_client.cc:647] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:53.913796 15642 tablet_copy_source_session.cc:205] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:53.914878 20522 tablet_copy_client.cc:523] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:53.918437 20522 tablet_bootstrap.cc:492] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:18:53.923141 20701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7082603ad2d6450e8863cd53b1897b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:53.923357 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:53.930163 20701 tablet_copy_client.cc:783] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:53.930495 20701 tablet_copy_client.cc:647] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:53.949542 20701 tablet_copy_client.cc:523] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:53.977821 20701 tablet_bootstrap.cc:492] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:18:53.981196 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240502 08:18:53.992851 12796 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 56921.1us
I20240502 08:18:53.994735 15622 raft_consensus.cc:1203] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:54.001034 20751 tablet_copy_client.cc:308] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:18:54.001770 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet c65f66921e73457eb07acec0670b2426 from peer 9333cb10227b4801b4fa7c24e22e131b ({username='slave'} at 127.6.5.7:49615)
I20240502 08:18:54.001945 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet c65f66921e73457eb07acec0670b2426 from peer 9333cb10227b4801b4fa7c24e22e131b at {username='slave'} at 127.6.5.7:49615: session id = 9333cb10227b4801b4fa7c24e22e131b-c65f66921e73457eb07acec0670b2426
I20240502 08:18:54.004602 15508 tablet_copy_source_session.cc:205] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:54.005829 20751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65f66921e73457eb07acec0670b2426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:54.011830 20522 tablet_bootstrap.cc:492] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:54.012217 20522 tablet_bootstrap.cc:492] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:18:54.012450 20522 ts_tablet_manager.cc:1367] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.094s user 0.049s sys 0.011s
I20240502 08:18:54.013134 20522 raft_consensus.cc:348] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:54.013331 20522 raft_consensus.cc:729] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:18:54.013530 20522 consensus_queue.cc:260] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:54.015455 20522 ts_tablet_manager.cc:1398] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:18:54.016561 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-5b29a9d8d26b4a388c89fb8e3a0fefe3 received from {username='slave'} at 127.6.5.6:55577
I20240502 08:18:54.016670 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-5b29a9d8d26b4a388c89fb8e3a0fefe3 on tablet 5b29a9d8d26b4a388c89fb8e3a0fefe3 with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:18:54.017921 20751 tablet_copy_client.cc:783] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:54.018167 20751 tablet_copy_client.cc:647] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:54.020593 20751 tablet_copy_client.cc:523] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:54.033670 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:54.039448 20751 tablet_bootstrap.cc:492] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:18:54.064528 20701 tablet_bootstrap.cc:492] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:54.065591 20701 tablet_bootstrap.cc:492] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4: Bootstrap complete.
I20240502 08:18:54.066437 20701 ts_tablet_manager.cc:1367] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.000s
I20240502 08:18:54.068403 20701 raft_consensus.cc:348] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:54.068625 20701 raft_consensus.cc:729] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:18:54.068804 20701 consensus_queue.cc:260] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:54.069975 20701 ts_tablet_manager.cc:1398] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:18:54.071498 15642 tablet_copy_service.cc:342] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Request end of tablet copy session 60b6e265e7754b7187307d05ef405dd4-7082603ad2d6450e8863cd53b1897b2e received from {username='slave'} at 127.6.5.3:49097
I20240502 08:18:54.071607 15642 tablet_copy_service.cc:433] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: ending tablet copy session 60b6e265e7754b7187307d05ef405dd4-7082603ad2d6450e8863cd53b1897b2e on tablet 7082603ad2d6450e8863cd53b1897b2e with peer 60b6e265e7754b7187307d05ef405dd4
I20240502 08:18:54.079807 20751 tablet_bootstrap.cc:492] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:54.080206 20751 tablet_bootstrap.cc:492] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap complete.
I20240502 08:18:54.084195 20751 ts_tablet_manager.cc:1367] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.001s
I20240502 08:18:54.084918 20751 raft_consensus.cc:348] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:54.085134 20751 raft_consensus.cc:729] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:18:54.085357 20751 consensus_queue.cc:260] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:54.085865 20751 ts_tablet_manager.cc:1398] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:18:54.088508 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session 9333cb10227b4801b4fa7c24e22e131b-c65f66921e73457eb07acec0670b2426 received from {username='slave'} at 127.6.5.7:49615
I20240502 08:18:54.088622 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session 9333cb10227b4801b4fa7c24e22e131b-c65f66921e73457eb07acec0670b2426 on tablet c65f66921e73457eb07acec0670b2426 with peer 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:18:54.109781 12861 raft_consensus.cc:1203] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:54.133008 15622 raft_consensus.cc:1203] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:18:54.222265 15754 raft_consensus.cc:1203] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:54.229501 12989 raft_consensus.cc:1203] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:18:54.604712 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:54.625094 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:54.639153 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:54.640600 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:54.674683 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:54.679092 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:54.690857 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:54.691450 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:54.893672 12727 consensus_queue.cc:237] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:54.895773 15622 raft_consensus.cc:1261] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:54.896137 12862 raft_consensus.cc:1261] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:54.896347 20655 consensus_queue.cc:1035] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:54.897106 20655 consensus_queue.cc:1035] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:54.899011 20858 raft_consensus.cc:2932] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:54.900655 12862 raft_consensus.cc:2932] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:54.902171 15622 raft_consensus.cc:2932] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:54.906196 12585 catalog_manager.cc:5484] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:54.921272 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b29a9d8d26b4a388c89fb8e3a0fefe3 with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:18:54.921841 20887 tablet_replica.cc:330] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:18:54.922108 20887 raft_consensus.cc:2229] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:54.922371 20887 raft_consensus.cc:2258] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:54.923070 20887 ts_tablet_manager.cc:1875] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:54.927536 20887 ts_tablet_manager.cc:1888] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:18:54.927685 20887 log.cc:1196] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/5b29a9d8d26b4a388c89fb8e3a0fefe3
I20240502 08:18:54.928426 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 5b29a9d8d26b4a388c89fb8e3a0fefe3 (table rebalance_test_table_5 [id=f63b67882a8d4f5d8817dc937654d88e]) successfully deleted
I20240502 08:18:55.167984 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:55.186616 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:55.219484 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:55.246157 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:55.249542 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:55.276932 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:55.332952 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:55.345286 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:55.496416 15754 consensus_queue.cc:237] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:18:55.498088 15620 raft_consensus.cc:1261] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:55.498566 12861 raft_consensus.cc:1261] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:55.499109 20112 consensus_queue.cc:1035] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:55.499408 20590 consensus_queue.cc:1035] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:55.501658 20112 raft_consensus.cc:2932] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:55.503227 12861 raft_consensus.cc:2932] T 6a9d9100dbaa4861ba1d2a1455f9756d P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:55.503508 15620 raft_consensus.cc:2932] T 6a9d9100dbaa4861ba1d2a1455f9756d P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:55.507083 12584 catalog_manager.cc:5484] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:18:55.517719 12708 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9d9100dbaa4861ba1d2a1455f9756d with delete_type TABLET_DATA_TOMBSTONED (TS 29b035434087435cbaf4ebcac98eedee not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40606
I20240502 08:18:55.518355 20967 tablet_replica.cc:330] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:18:55.518611 20967 raft_consensus.cc:2229] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:55.518837 20967 raft_consensus.cc:2258] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:55.519491 20967 ts_tablet_manager.cc:1875] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:55.526250 20967 ts_tablet_manager.cc:1888] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:55.526417 20967 log.cc:1196] T 6a9d9100dbaa4861ba1d2a1455f9756d P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/6a9d9100dbaa4861ba1d2a1455f9756d
I20240502 08:18:55.527417 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 6a9d9100dbaa4861ba1d2a1455f9756d (table rebalancer_extra_table_38 [id=52eb1ee601914170a78e4182cb32211c]) successfully deleted
I20240502 08:18:55.799270 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:55.827178 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:55.854844 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:55.861173 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:55.884634 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:55.889879 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:55.919410 15468 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:55.935830 12842 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:18:56.022289 15486 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c65f66921e73457eb07acec0670b2426"
dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34020
I20240502 08:18:56.022550 15486 raft_consensus.cc:595] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:56.127023 20576 raft_consensus.cc:979] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3: : Instructing follower e1dd14f7ca764a1490df419f13f2804a to start an election
I20240502 08:18:56.127161 20576 raft_consensus.cc:1067] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Signalling peer e1dd14f7ca764a1490df419f13f2804a to start an election
I20240502 08:18:56.128077 12861 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c65f66921e73457eb07acec0670b2426"
dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
from {username='slave'} at 127.6.5.5:45483
I20240502 08:18:56.128335 12861 raft_consensus.cc:482] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:56.128448 12861 raft_consensus.cc:3037] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:56.139405 12861 raft_consensus.cc:504] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:56.141005 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f66921e73457eb07acec0670b2426" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:56.141198 15486 raft_consensus.cc:3032] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:56.141288 15486 raft_consensus.cc:729] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:56.141566 15486 consensus_queue.cc:260] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:56.141918 15486 raft_consensus.cc:3037] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:56.143698 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f66921e73457eb07acec0670b2426" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:18:56.143844 15753 raft_consensus.cc:3037] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:56.143849 12991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65f66921e73457eb07acec0670b2426" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:18:56.144027 12991 raft_consensus.cc:3037] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:56.144819 15486 raft_consensus.cc:2454] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 2.
I20240502 08:18:56.145931 15753 raft_consensus.cc:2454] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 2.
I20240502 08:18:56.146385 12991 raft_consensus.cc:2454] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 2.
I20240502 08:18:56.146869 12796 leader_election.cc:302] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, db7893bb29a047b3b82cc60bfd8930a3, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:18:56.147478 20776 raft_consensus.cc:2781] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:56.147614 20776 raft_consensus.cc:686] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 2 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:18:56.147869 20776 consensus_queue.cc:237] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:18:56.150277 12861 leader_election.cc:288] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:18:56.153085 12586 catalog_manager.cc:5484] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 1 to 2, leader changed from db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 2 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:18:56.298822 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:56.318346 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:56.319789 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:56.340840 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:56.411636 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:56.426110 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:56.468580 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:18:56.483086 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:56.560379 15753 raft_consensus.cc:1261] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:18:56.561262 20775 consensus_queue.cc:1035] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:56.709182 15486 raft_consensus.cc:1261] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:18:56.709769 21072 consensus_queue.cc:1035] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:56.722766 12991 raft_consensus.cc:1261] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 2 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:18:56.723299 21072 consensus_queue.cc:1035] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:56.922685 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:56.950717 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:56.984196 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:56.984652 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:57.003687 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:57.017628 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:57.027089 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:57.067899 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:18:57.202821 15754 consensus_queue.cc:237] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:18:57.204769 15621 raft_consensus.cc:1261] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:57.204998 12991 raft_consensus.cc:1261] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:18:57.205420 20112 consensus_queue.cc:1035] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:57.205677 20590 consensus_queue.cc:1035] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:57.209213 20112 raft_consensus.cc:2932] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:57.223425 12580 catalog_manager.cc:5484] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:57.225159 15621 raft_consensus.cc:2932] T 18c58e2a9a814bcb8ff6bb9e3011084f P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:57.231755 12991 raft_consensus.cc:2932] T 18c58e2a9a814bcb8ff6bb9e3011084f P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:18:57.244004 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 18c58e2a9a814bcb8ff6bb9e3011084f with delete_type TABLET_DATA_TOMBSTONED (TS 10f0e36203914b85b77e78609ffced98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47272
I20240502 08:18:57.244581 21219 tablet_replica.cc:330] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:18:57.244771 21219 raft_consensus.cc:2229] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:57.244943 21219 raft_consensus.cc:2258] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:57.245448 21219 ts_tablet_manager.cc:1875] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:57.251240 21219 ts_tablet_manager.cc:1888] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:18:57.251405 21219 log.cc:1196] T 18c58e2a9a814bcb8ff6bb9e3011084f P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/18c58e2a9a814bcb8ff6bb9e3011084f
I20240502 08:18:57.252069 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 18c58e2a9a814bcb8ff6bb9e3011084f (table rebalancer_extra_table_6 [id=f19b8479d1b74f17b2ee9e2117749102]) successfully deleted
I20240502 08:18:57.550076 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:57.566956 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:57.607940 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:57.611150 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:57.622897 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:57.648169 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:57.719671 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:18:57.748858 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:57.870997 12991 consensus_queue.cc:237] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:57.872709 15622 raft_consensus.cc:1261] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:57.872715 13121 raft_consensus.cc:1261] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:57.873284 20277 consensus_queue.cc:1035] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:57.873703 20274 consensus_queue.cc:1035] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:57.876360 20277 raft_consensus.cc:2932] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:57.877171 15622 raft_consensus.cc:2932] T 83e0c78cb2374ba483315ca8a827e532 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:57.881868 12584 catalog_manager.cc:5484] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:57.895793 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet 83e0c78cb2374ba483315ca8a827e532 with delete_type TABLET_DATA_TOMBSTONED (TS e1dd14f7ca764a1490df419f13f2804a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:57.896207 13121 raft_consensus.cc:2932] T 83e0c78cb2374ba483315ca8a827e532 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:57.896515 21304 tablet_replica.cc:330] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:57.896767 21304 raft_consensus.cc:2229] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:57.896996 21304 raft_consensus.cc:2258] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:57.897751 21304 ts_tablet_manager.cc:1875] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:57.905267 21304 ts_tablet_manager.cc:1888] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:18:57.905459 21304 log.cc:1196] T 83e0c78cb2374ba483315ca8a827e532 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/83e0c78cb2374ba483315ca8a827e532
I20240502 08:18:57.906119 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 83e0c78cb2374ba483315ca8a827e532 (table rebalance_test_table_1 [id=eafcd56a22d944b391206b320ed18540]) successfully deleted
I20240502 08:18:58.174441 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:58.209679 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:58.249265 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:58.250085 12708 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:58.314182 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:58.330138 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:58.337013 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:18:58.377628 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:18:58.494833 15488 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3b930dcb3863402296a205fdb65d6fc3"
dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34068
I20240502 08:18:58.495007 15488 raft_consensus.cc:595] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:58.512902 21183 raft_consensus.cc:979] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3: : Instructing follower 29b035434087435cbaf4ebcac98eedee to start an election
I20240502 08:18:58.513026 21183 raft_consensus.cc:1067] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Signalling peer 29b035434087435cbaf4ebcac98eedee to start an election
I20240502 08:18:58.513805 12730 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b930dcb3863402296a205fdb65d6fc3"
dest_uuid: "29b035434087435cbaf4ebcac98eedee"
from {username='slave'} at 127.6.5.5:46815
I20240502 08:18:58.514039 12730 raft_consensus.cc:482] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:58.514147 12730 raft_consensus.cc:3037] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:58.517791 12730 raft_consensus.cc:504] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:58.521049 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b930dcb3863402296a205fdb65d6fc3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:18:58.521246 15486 raft_consensus.cc:3032] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:58.521342 15486 raft_consensus.cc:729] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:18:58.521611 15486 consensus_queue.cc:260] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:58.522035 15486 raft_consensus.cc:3037] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:58.523250 12730 leader_election.cc:288] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:18:58.524583 15486 raft_consensus.cc:2454] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 2.
I20240502 08:18:58.525192 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b930dcb3863402296a205fdb65d6fc3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:18:58.525405 13122 raft_consensus.cc:3037] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:58.528010 13122 raft_consensus.cc:2454] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 2.
I20240502 08:18:58.529230 12663 leader_election.cc:302] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, 6a0b05f3250d415590b0edf6b44fbe1f, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:18:58.529845 12861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b930dcb3863402296a205fdb65d6fc3" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:18:58.530042 12861 raft_consensus.cc:3037] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:58.530987 20858 raft_consensus.cc:2781] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:58.531103 20858 raft_consensus.cc:686] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [term 2 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:18:58.531350 20858 consensus_queue.cc:237] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:18:58.532729 12861 raft_consensus.cc:2454] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 2.
I20240502 08:18:58.538194 12584 catalog_manager.cc:5484] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 1 to 2, leader changed from db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 2 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
W20240502 08:18:58.722962 12664 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 47507.6us
I20240502 08:18:58.788936 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:58.807024 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:58.826313 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:58.834765 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:58.839408 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:58.848623 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:58.894390 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:18:58.942391 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:18:58.963454 15486 raft_consensus.cc:1261] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:18:58.964104 21409 consensus_queue.cc:1035] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:59.094192 13122 raft_consensus.cc:1261] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:18:59.094797 21409 consensus_queue.cc:1035] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:59.111795 12862 raft_consensus.cc:1261] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:18:59.112366 21409 consensus_queue.cc:1035] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:18:59.276136 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:59.304033 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:59.304420 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:59.323736 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:59.364491 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:59.374356 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:18:59.400885 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:59.419143 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:18:59.558034 13121 consensus_queue.cc:237] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:18:59.559959 15622 raft_consensus.cc:1261] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:59.560221 12730 raft_consensus.cc:1261] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:59.560773 19785 consensus_queue.cc:1035] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:59.561049 21494 consensus_queue.cc:1035] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:59.564065 21033 raft_consensus.cc:2932] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:59.564622 15622 raft_consensus.cc:2932] T 7a8e5440619a4cb0b40dd286862184cd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:59.569207 12730 raft_consensus.cc:2932] T 7a8e5440619a4cb0b40dd286862184cd P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:59.569631 12586 catalog_manager.cc:5484] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:18:59.580289 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a8e5440619a4cb0b40dd286862184cd with delete_type TABLET_DATA_TOMBSTONED (TS e1dd14f7ca764a1490df419f13f2804a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56258
I20240502 08:18:59.581519 21517 tablet_replica.cc:330] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:18:59.581885 21517 raft_consensus.cc:2229] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:59.582151 21517 raft_consensus.cc:2258] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:59.582911 21517 ts_tablet_manager.cc:1875] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:59.588084 21517 ts_tablet_manager.cc:1888] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:18:59.588196 21517 log.cc:1196] T 7a8e5440619a4cb0b40dd286862184cd P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/7a8e5440619a4cb0b40dd286862184cd
I20240502 08:18:59.590255 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 7a8e5440619a4cb0b40dd286862184cd (table rebalance_test_table_15 [id=f13587d289664cc99887ffe5c7242379]) successfully deleted
I20240502 08:18:59.876592 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:59.901911 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:59.912528 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:18:59.924423 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:18:59.939538 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:18:59.967713 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:18:59.978283 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:18:59.990698 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:00.125661 12861 consensus_queue.cc:237] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:00.127470 15754 raft_consensus.cc:1261] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:00.127681 12991 raft_consensus.cc:1261] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:00.127998 21442 consensus_queue.cc:1035] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:00.128271 21443 consensus_queue.cc:1035] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:00.131115 21442 raft_consensus.cc:2932] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:00.131527 12991 raft_consensus.cc:2932] T 5be94bf194b04e1d965274a968f30fbf P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:00.132128 15754 raft_consensus.cc:2932] T 5be94bf194b04e1d965274a968f30fbf P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:00.137503 12585 catalog_manager.cc:5484] T 5be94bf194b04e1d965274a968f30fbf P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:00.145815 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be94bf194b04e1d965274a968f30fbf with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:19:00.146556 21605 tablet_replica.cc:330] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:19:00.146872 21605 raft_consensus.cc:2229] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:00.147188 21605 raft_consensus.cc:2258] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:00.147971 21605 ts_tablet_manager.cc:1875] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:00.153528 21605 ts_tablet_manager.cc:1888] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:00.153611 21605 log.cc:1196] T 5be94bf194b04e1d965274a968f30fbf P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/5be94bf194b04e1d965274a968f30fbf
I20240502 08:19:00.154186 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 5be94bf194b04e1d965274a968f30fbf (table rebalance_test_table_5 [id=f63b67882a8d4f5d8817dc937654d88e]) successfully deleted
I20240502 08:19:00.445206 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:00.465431 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:00.494684 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:00.525607 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:00.546689 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:00.665633 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:00.693985 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:00.721685 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:00.819046 13122 consensus_queue.cc:237] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:00.821033 15887 raft_consensus.cc:1261] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:00.821241 12989 raft_consensus.cc:1261] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:00.821671 21033 consensus_queue.cc:1035] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:00.822106 21033 consensus_queue.cc:1035] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:00.824216 21494 raft_consensus.cc:2932] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:00.824615 15887 raft_consensus.cc:2932] T 109bc82d7ff94f059dcbc80555b88b1b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:00.831408 12586 catalog_manager.cc:5484] T 109bc82d7ff94f059dcbc80555b88b1b P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:00.837466 12989 raft_consensus.cc:2932] T 109bc82d7ff94f059dcbc80555b88b1b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:00.862726 12708 tablet_service.cc:1514] Processing DeleteTablet for tablet 109bc82d7ff94f059dcbc80555b88b1b with delete_type TABLET_DATA_TOMBSTONED (TS 29b035434087435cbaf4ebcac98eedee not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40606
I20240502 08:19:00.863268 21706 tablet_replica.cc:330] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:19:00.863461 21706 raft_consensus.cc:2229] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:00.863731 21706 raft_consensus.cc:2258] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:00.864730 21706 ts_tablet_manager.cc:1875] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:00.870676 21706 ts_tablet_manager.cc:1888] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:00.870796 21706 log.cc:1196] T 109bc82d7ff94f059dcbc80555b88b1b P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/109bc82d7ff94f059dcbc80555b88b1b
I20240502 08:19:00.871487 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 109bc82d7ff94f059dcbc80555b88b1b (table rebalancer_extra_table_52 [id=b77da14560704c279516343b861254e7]) successfully deleted
I20240502 08:19:01.074385 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:01.108831 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:01.121371 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:01.130632 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:01.154017 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:01.174980 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:01.205185 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:01.208964 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:01.319907 15885 consensus_queue.cc:237] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:01.322119 15622 raft_consensus.cc:1261] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:01.322280 15754 raft_consensus.cc:1261] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:01.322705 21715 consensus_queue.cc:1035] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:01.322994 21714 consensus_queue.cc:1035] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:01.325407 21715 raft_consensus.cc:2932] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:01.326001 15622 raft_consensus.cc:2932] T c60b24cf2d4740ae9b8829e288bc2530 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:01.333266 12586 catalog_manager.cc:5484] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:01.339068 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet c60b24cf2d4740ae9b8829e288bc2530 with delete_type TABLET_DATA_TOMBSTONED (TS db7893bb29a047b3b82cc60bfd8930a3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46880
I20240502 08:19:01.339957 21779 tablet_replica.cc:330] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:19:01.340305 21779 raft_consensus.cc:2229] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:01.340740 21779 raft_consensus.cc:2258] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:01.341408 21779 ts_tablet_manager.cc:1875] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:01.350113 21779 ts_tablet_manager.cc:1888] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:01.349857 15754 raft_consensus.cc:2932] T c60b24cf2d4740ae9b8829e288bc2530 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:01.350797 21779 log.cc:1196] T c60b24cf2d4740ae9b8829e288bc2530 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/c60b24cf2d4740ae9b8829e288bc2530
I20240502 08:19:01.351555 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet c60b24cf2d4740ae9b8829e288bc2530 (table rebalancer_extra_table_26 [id=602cdebb879f43d3b1a9666d8bd7cddc]) successfully deleted
I20240502 08:19:01.645936 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:01.685353 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:01.723037 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:01.731633 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:01.789278 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:01.844877 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:01.856881 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:01.909271 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:01.992573 12989 consensus_queue.cc:237] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:01.994988 15885 raft_consensus.cc:1261] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:01.995472 13122 raft_consensus.cc:1261] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:01.996060 20277 consensus_queue.cc:1035] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:01.996327 20274 consensus_queue.cc:1035] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:01.998778 20277 raft_consensus.cc:2932] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:01.999473 15885 raft_consensus.cc:2932] T a3583c839d364bae819928eca86010ec P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:02.008198 12588 catalog_manager.cc:5484] T a3583c839d364bae819928eca86010ec P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:02.020399 13121 raft_consensus.cc:2932] T a3583c839d364bae819928eca86010ec P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
W20240502 08:19:02.060137 15818 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.7:38851, user_credentials={real_user=slave}} blocked reactor thread for 45690us
I20240502 08:19:02.063843 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet a3583c839d364bae819928eca86010ec with delete_type TABLET_DATA_TOMBSTONED (TS e1dd14f7ca764a1490df419f13f2804a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56258
I20240502 08:19:02.064484 21871 tablet_replica.cc:330] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:19:02.064819 21871 raft_consensus.cc:2229] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:02.065207 21871 raft_consensus.cc:2258] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:02.066371 21871 ts_tablet_manager.cc:1875] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:02.071852 21871 ts_tablet_manager.cc:1888] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:02.071987 21871 log.cc:1196] T a3583c839d364bae819928eca86010ec P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/a3583c839d364bae819928eca86010ec
I20240502 08:19:02.073119 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet a3583c839d364bae819928eca86010ec (table rebalance_test_table_11 [id=02e24b5edb434940bb115fce3e1947fb]) successfully deleted
I20240502 08:19:02.332811 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:02.371191 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:02.374917 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:02.416568 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:02.416743 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:02.423022 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:02.517534 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:02.533722 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:02.790997 15885 consensus_queue.cc:237] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:02.793037 13122 raft_consensus.cc:1261] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:02.793038 15622 raft_consensus.cc:1261] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:02.793635 21867 consensus_queue.cc:1035] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:02.793901 21715 consensus_queue.cc:1035] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:02.796990 21867 raft_consensus.cc:2932] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:02.800552 13122 raft_consensus.cc:2932] T 0e882f58c7f344f7929b8a30bd65fa44 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:02.803328 12587 catalog_manager.cc:5484] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:02.845805 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e882f58c7f344f7929b8a30bd65fa44 with delete_type TABLET_DATA_TOMBSTONED (TS 9333cb10227b4801b4fa7c24e22e131b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45134
I20240502 08:19:02.848718 15621 raft_consensus.cc:2932] T 0e882f58c7f344f7929b8a30bd65fa44 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:02.874413 21957 tablet_replica.cc:330] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:19:02.874775 21957 raft_consensus.cc:2229] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:02.875073 21957 raft_consensus.cc:2258] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:02.875754 21957 ts_tablet_manager.cc:1875] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:02.884685 21957 ts_tablet_manager.cc:1888] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:02.885339 21957 log.cc:1196] T 0e882f58c7f344f7929b8a30bd65fa44 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/0e882f58c7f344f7929b8a30bd65fa44
I20240502 08:19:02.887113 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 0e882f58c7f344f7929b8a30bd65fa44 (table rebalancer_extra_table_36 [id=b9146919078d47a99126e22456f8f239]) successfully deleted
I20240502 08:19:03.160542 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:03.162601 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:03.191233 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:03.215068 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:03.266484 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:03.277024 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:03.318620 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:03.331734 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:03.465718 12730 consensus_queue.cc:237] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:03.478740 12991 raft_consensus.cc:1261] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:03.478986 15753 raft_consensus.cc:1261] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:03.479830 20254 consensus_queue.cc:1035] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:03.480568 20254 consensus_queue.cc:1035] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:03.483263 20254 raft_consensus.cc:2932] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:03.484012 15753 raft_consensus.cc:2932] T b87724cd8beb4449b272f972c85a6666 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:03.490090 12991 raft_consensus.cc:2932] T b87724cd8beb4449b272f972c85a6666 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:03.491046 12580 catalog_manager.cc:5484] T b87724cd8beb4449b272f972c85a6666 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:03.497845 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet b87724cd8beb4449b272f972c85a6666 with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:19:03.500440 22051 tablet_replica.cc:330] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:19:03.500715 22051 raft_consensus.cc:2229] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:03.501035 22051 raft_consensus.cc:2258] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:03.502061 22051 ts_tablet_manager.cc:1875] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:03.508425 22051 ts_tablet_manager.cc:1888] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:03.508587 22051 log.cc:1196] T b87724cd8beb4449b272f972c85a6666 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/b87724cd8beb4449b272f972c85a6666
I20240502 08:19:03.509313 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet b87724cd8beb4449b272f972c85a6666 (table rebalance_test_table_13 [id=f2c6e6573e254b7d815efdcd481a81a2]) successfully deleted
I20240502 08:19:03.855119 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:03.855304 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:03.856832 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:03.887539 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:03.898686 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:03.931902 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:03.969624 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:03.970428 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:04.088956 12727 consensus_queue.cc:237] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:04.091076 15753 raft_consensus.cc:1261] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:04.091401 12989 raft_consensus.cc:1261] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:04.091857 20254 consensus_queue.cc:1035] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:04.092115 22116 consensus_queue.cc:1035] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:04.094913 20254 raft_consensus.cc:2932] T a542eb208b254055b02765aceba5b405 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:04.098104 12989 raft_consensus.cc:2932] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:04.104456 12581 catalog_manager.cc:5484] T a542eb208b254055b02765aceba5b405 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:04.105026 15753 raft_consensus.cc:2932] T a542eb208b254055b02765aceba5b405 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:04.123342 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet a542eb208b254055b02765aceba5b405 with delete_type TABLET_DATA_TOMBSTONED (TS 10f0e36203914b85b77e78609ffced98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47272
I20240502 08:19:04.139135 22143 tablet_replica.cc:330] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:19:04.139763 22143 raft_consensus.cc:2229] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:04.139994 22143 raft_consensus.cc:2258] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:04.140625 22143 ts_tablet_manager.cc:1875] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:04.146100 22143 ts_tablet_manager.cc:1888] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:04.146215 22143 log.cc:1196] T a542eb208b254055b02765aceba5b405 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/a542eb208b254055b02765aceba5b405
I20240502 08:19:04.146807 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet a542eb208b254055b02765aceba5b405 (table rebalancer_extra_table_4 [id=c68337015ae744879e3d0cbe8455502e]) successfully deleted
I20240502 08:19:04.394650 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:04.400406 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:04.427654 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:04.442057 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:04.466737 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:04.475111 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:04.563688 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:04.601531 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:04.739624 15754 consensus_queue.cc:237] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:04.741770 15887 raft_consensus.cc:1261] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:04.741792 12727 raft_consensus.cc:1261] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:04.742295 22026 consensus_queue.cc:1035] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:04.742837 21576 consensus_queue.cc:1035] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:04.745026 22026 raft_consensus.cc:2932] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:04.746835 12727 raft_consensus.cc:2932] T b73852b46f7e4db89e30d83edafa6413 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:04.750536 15887 raft_consensus.cc:2932] T b73852b46f7e4db89e30d83edafa6413 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:04.751943 12588 catalog_manager.cc:5484] T b73852b46f7e4db89e30d83edafa6413 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:04.768891 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet b73852b46f7e4db89e30d83edafa6413 with delete_type TABLET_DATA_TOMBSTONED (TS 60b6e265e7754b7187307d05ef405dd4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38740
I20240502 08:19:04.769470 22224 tablet_replica.cc:330] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:19:04.769775 22224 raft_consensus.cc:2229] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:04.770061 22224 raft_consensus.cc:2258] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:04.770921 22224 ts_tablet_manager.cc:1875] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:04.779192 22224 ts_tablet_manager.cc:1888] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:04.779362 22224 log.cc:1196] T b73852b46f7e4db89e30d83edafa6413 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/b73852b46f7e4db89e30d83edafa6413
I20240502 08:19:04.780042 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet b73852b46f7e4db89e30d83edafa6413 (table rebalancer_extra_table_54 [id=b251c3886b9548c2bbc30da21fc3f84b]) successfully deleted
I20240502 08:19:05.025563 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:05.062127 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:05.073343 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:05.125308 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:05.125703 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:05.139811 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:05.160768 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:05.184727 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:05.307940 12861 consensus_queue.cc:237] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:05.310429 15488 raft_consensus.cc:1261] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:05.310428 12991 raft_consensus.cc:1261] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:05.311371 21442 consensus_queue.cc:1035] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:05.311864 21442 consensus_queue.cc:1035] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:05.313652 21442 raft_consensus.cc:2932] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:05.314226 12991 raft_consensus.cc:2932] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:05.318315 15488 raft_consensus.cc:2932] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:05.319437 12584 catalog_manager.cc:5484] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:05.334270 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e3cc08d9534400ab7ee8db08e6e7ee0 with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:05.349905 22307 tablet_replica.cc:330] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:05.350217 22307 raft_consensus.cc:2229] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:05.350422 22307 raft_consensus.cc:2258] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:05.351040 22307 ts_tablet_manager.cc:1875] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:05.356688 22307 ts_tablet_manager.cc:1888] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:05.356806 22307 log.cc:1196] T 2e3cc08d9534400ab7ee8db08e6e7ee0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/2e3cc08d9534400ab7ee8db08e6e7ee0
I20240502 08:19:05.359838 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 2e3cc08d9534400ab7ee8db08e6e7ee0 (table rebalancer_extra_table_62 [id=6aba7496ff9b4c90ae1f70fee8b918e0]) successfully deleted
I20240502 08:19:05.637763 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:05.666129 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:05.678967 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:05.684240 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:05.687783 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:05.689905 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:05.723840 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:05.776654 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:05.927971 12991 consensus_queue.cc:237] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:05.929746 13122 raft_consensus.cc:1261] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:05.930207 15885 raft_consensus.cc:1261] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:05.930269 20277 consensus_queue.cc:1035] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:05.930783 20277 consensus_queue.cc:1035] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:05.933226 22282 raft_consensus.cc:2932] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:05.933866 13122 raft_consensus.cc:2932] T 0fa42a21c83942f789dcebed741b52ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:05.938697 12588 catalog_manager.cc:5484] T 0fa42a21c83942f789dcebed741b52ae P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:05.948755 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fa42a21c83942f789dcebed741b52ae with delete_type TABLET_DATA_TOMBSTONED (TS e1dd14f7ca764a1490df419f13f2804a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56258
I20240502 08:19:05.949440 22393 tablet_replica.cc:330] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:19:05.949725 22393 raft_consensus.cc:2229] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:05.950176 22393 raft_consensus.cc:2258] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:05.951121 22393 ts_tablet_manager.cc:1875] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:05.957906 22393 ts_tablet_manager.cc:1888] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:05.958082 22393 log.cc:1196] T 0fa42a21c83942f789dcebed741b52ae P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/0fa42a21c83942f789dcebed741b52ae
I20240502 08:19:05.958833 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 0fa42a21c83942f789dcebed741b52ae (table rebalance_test_table_5 [id=f63b67882a8d4f5d8817dc937654d88e]) successfully deleted
I20240502 08:19:05.964183 15885 raft_consensus.cc:2932] T 0fa42a21c83942f789dcebed741b52ae P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:06.183257 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:06.203449 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:06.225140 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:06.241323 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:06.313472 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:06.326707 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:06.353713 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:06.361382 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:06.510564 12727 consensus_queue.cc:237] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:06.512454 15753 raft_consensus.cc:1261] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:06.512755 12991 raft_consensus.cc:1261] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:06.513137 22116 consensus_queue.cc:1035] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:06.513418 22200 consensus_queue.cc:1035] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:06.515930 22116 raft_consensus.cc:2932] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:06.516562 15753 raft_consensus.cc:2932] T f1581261b4c144a9b670786eb3d26466 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:06.520303 12991 raft_consensus.cc:2932] T f1581261b4c144a9b670786eb3d26466 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:06.521939 12582 catalog_manager.cc:5484] T f1581261b4c144a9b670786eb3d26466 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:06.529331 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet f1581261b4c144a9b670786eb3d26466 with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:19:06.529867 22470 tablet_replica.cc:330] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:19:06.530053 22470 raft_consensus.cc:2229] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:06.530268 22470 raft_consensus.cc:2258] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:06.531497 22470 ts_tablet_manager.cc:1875] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:06.537318 22470 ts_tablet_manager.cc:1888] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:06.537431 22470 log.cc:1196] T f1581261b4c144a9b670786eb3d26466 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/f1581261b4c144a9b670786eb3d26466
I20240502 08:19:06.538141 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet f1581261b4c144a9b670786eb3d26466 (table rebalance_test_table_19 [id=c7115c9eea2a450b944a15086ffe02bd]) successfully deleted
I20240502 08:19:06.796694 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:06.817214 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:06.834712 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:06.866339 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:06.883277 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:06.906495 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:06.938421 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:06.938429 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:07.049224 15621 consensus_queue.cc:237] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:07.051213 13122 raft_consensus.cc:1261] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:07.051247 12991 raft_consensus.cc:1261] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:07.051810 22188 consensus_queue.cc:1035] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:07.052299 22188 consensus_queue.cc:1035] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:07.056313 22188 raft_consensus.cc:2932] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:07.056980 13122 raft_consensus.cc:2932] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:07.062034 12581 catalog_manager.cc:5484] T 7082603ad2d6450e8863cd53b1897b2e P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:07.071271 12991 raft_consensus.cc:2932] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:07.095058 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 7082603ad2d6450e8863cd53b1897b2e with delete_type TABLET_DATA_TOMBSTONED (TS 10f0e36203914b85b77e78609ffced98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47272
I20240502 08:19:07.103353 22563 tablet_replica.cc:330] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:19:07.103540 22563 raft_consensus.cc:2229] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:07.103719 22563 raft_consensus.cc:2258] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:07.104194 22563 ts_tablet_manager.cc:1875] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:07.108762 22563 ts_tablet_manager.cc:1888] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:07.108850 22563 log.cc:1196] T 7082603ad2d6450e8863cd53b1897b2e P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/7082603ad2d6450e8863cd53b1897b2e
I20240502 08:19:07.110424 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 7082603ad2d6450e8863cd53b1897b2e (table rebalancer_extra_table_34 [id=863bf9b609384908ae899e09f3d2e04c]) successfully deleted
I20240502 08:19:07.405040 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:07.406111 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:07.443501 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:07.443639 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:07.448133 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:07.473284 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:07.473711 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:07.534675 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:07.663496 12861 consensus_queue.cc:237] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:07.665764 12991 raft_consensus.cc:1261] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 2 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:07.666200 22195 consensus_queue.cc:1035] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:07.668658 21442 raft_consensus.cc:2932] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:07.669277 12991 raft_consensus.cc:2932] T c65f66921e73457eb07acec0670b2426 P 60b6e265e7754b7187307d05ef405dd4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:07.671213 15753 raft_consensus.cc:1261] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:07.671725 21442 consensus_queue.cc:1035] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:07.674681 12581 catalog_manager.cc:5484] T c65f66921e73457eb07acec0670b2426 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index 2 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New cstate: current_term: 2 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:07.697190 15753 raft_consensus.cc:2932] T c65f66921e73457eb07acec0670b2426 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:07.707450 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet c65f66921e73457eb07acec0670b2426 with delete_type TABLET_DATA_TOMBSTONED (TS db7893bb29a047b3b82cc60bfd8930a3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46880
I20240502 08:19:07.718083 22658 tablet_replica.cc:330] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:19:07.718323 22658 raft_consensus.cc:2229] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:07.720003 22658 raft_consensus.cc:2258] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:07.720543 22658 ts_tablet_manager.cc:1875] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:07.724999 22658 ts_tablet_manager.cc:1888] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 08:19:07.725138 22658 log.cc:1196] T c65f66921e73457eb07acec0670b2426 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/c65f66921e73457eb07acec0670b2426
I20240502 08:19:07.725811 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet c65f66921e73457eb07acec0670b2426 (table rebalancer_extra_table_60 [id=94d0e1dcc3fb43e694fe94acec28e365]) successfully deleted
I20240502 08:19:07.969368 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:07.992947 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:08.004444 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:08.007946 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:08.015588 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:08.016453 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:08.065135 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:08.073228 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:08.225214 12730 consensus_queue.cc:237] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:08.227519 13121 raft_consensus.cc:1261] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:08.228061 12861 raft_consensus.cc:1261] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:08.228873 22200 consensus_queue.cc:1035] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:08.229337 22200 consensus_queue.cc:1035] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:08.233481 12860 raft_consensus.cc:2932] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:08.233104 22116 raft_consensus.cc:2932] T 3b930dcb3863402296a205fdb65d6fc3 P 29b035434087435cbaf4ebcac98eedee [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:08.234266 13121 raft_consensus.cc:2932] T 3b930dcb3863402296a205fdb65d6fc3 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:08.239112 12588 catalog_manager.cc:5484] T 3b930dcb3863402296a205fdb65d6fc3 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index 2 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New cstate: current_term: 2 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:08.256846 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b930dcb3863402296a205fdb65d6fc3 with delete_type TABLET_DATA_TOMBSTONED (TS db7893bb29a047b3b82cc60bfd8930a3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46880
I20240502 08:19:08.258174 22754 tablet_replica.cc:330] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:19:08.258644 22754 raft_consensus.cc:2229] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:08.258894 22754 raft_consensus.cc:2258] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:08.259589 22754 ts_tablet_manager.cc:1875] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:08.265849 22754 ts_tablet_manager.cc:1888] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 08:19:08.266009 22754 log.cc:1196] T 3b930dcb3863402296a205fdb65d6fc3 P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/3b930dcb3863402296a205fdb65d6fc3
I20240502 08:19:08.266737 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 3b930dcb3863402296a205fdb65d6fc3 (table rebalancer_extra_table_34 [id=863bf9b609384908ae899e09f3d2e04c]) successfully deleted
I20240502 08:19:08.430603 15753 consensus_queue.cc:237] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:08.433014 15886 raft_consensus.cc:1261] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:19:08.433091 15622 raft_consensus.cc:1261] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 2 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:19:08.433683 21576 consensus_queue.cc:1035] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:08.434509 21576 consensus_queue.cc:1035] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:19:08.439468 15688 consensus_peers.cc:485] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 040878c4c4ae4684b02296ee9eac8985. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:08.462100 21576 raft_consensus.cc:2932] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:08.462713 15886 raft_consensus.cc:2932] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:08.469444 12587 catalog_manager.cc:5484] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 2 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:08.470244 15621 raft_consensus.cc:2932] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:08.536867 12992 consensus_queue.cc:237] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:08.538812 13121 raft_consensus.cc:1261] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:08.538949 12861 raft_consensus.cc:1261] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:08.539482 20277 consensus_queue.cc:1035] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:08.539855 22635 consensus_queue.cc:1035] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:19:08.540589 12927 consensus_peers.cc:485] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Couldn't send request to peer 10f0e36203914b85b77e78609ffced98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 660b389a179f49eeb3007dea6ce078e9. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:08.544092 22845 raft_consensus.cc:2932] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:08.548465 12860 raft_consensus.cc:2932] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:08.549561 13121 raft_consensus.cc:2932] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:08.554944 12579 catalog_manager.cc:5484] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:08.597524 12730 consensus_queue.cc:237] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:08.604000 12861 raft_consensus.cc:1261] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:08.604580 12992 raft_consensus.cc:1261] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:08.604709 22745 consensus_queue.cc:1035] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:19:08.605648 12664 consensus_peers.cc:485] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee -> Peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Couldn't send request to peer db7893bb29a047b3b82cc60bfd8930a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f07b2caf1754c469a8e33382f615019. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:08.605947 22745 consensus_queue.cc:1035] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:08.610791 22745 raft_consensus.cc:2932] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:08.611356 12861 raft_consensus.cc:2932] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:08.615001 12992 raft_consensus.cc:2932] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:08.617689 12588 catalog_manager.cc:5484] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:08.668027 12992 consensus_queue.cc:237] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:08.676077 12730 raft_consensus.cc:1261] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:08.676329 13121 raft_consensus.cc:1261] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:08.676391 12925 consensus_peers.cc:485] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 -> Peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Couldn't send request to peer 9333cb10227b4801b4fa7c24e22e131b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5be106986ad247c597666d46a1dede7d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:08.676581 22849 consensus_queue.cc:1035] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:08.676841 22845 consensus_queue.cc:1035] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:08.694134 22845 raft_consensus.cc:2932] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:08.698417 13121 raft_consensus.cc:2932] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:08.698563 12730 raft_consensus.cc:2932] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:08.701023 12587 catalog_manager.cc:5484] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:08.740206 13121 consensus_queue.cc:237] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:08.742823 15621 raft_consensus.cc:1261] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:08.744486 22518 consensus_queue.cc:1035] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:08.745815 12992 raft_consensus.cc:1261] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:08.747268 13056 consensus_peers.cc:485] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 087809a059d14b1f9a24ec3e956e1305. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:08.751680 22651 consensus_queue.cc:1035] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:08.763690 22651 raft_consensus.cc:2932] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:08.768877 15622 raft_consensus.cc:2932] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:08.768713 12992 raft_consensus.cc:2932] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:08.773553 12579 catalog_manager.cc:5484] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:08.821664 12860 consensus_queue.cc:237] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:08.824599 13121 raft_consensus.cc:1261] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:08.824808 12991 raft_consensus.cc:1261] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:08.825048 22910 consensus_queue.cc:1035] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:08.825300 22195 consensus_queue.cc:1035] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:19:08.825515 12797 consensus_peers.cc:485] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a -> Peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Couldn't send request to peer 10f0e36203914b85b77e78609ffced98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 581d2ed76e2e446eb84b94231ebb97f4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:08.833187 22910 raft_consensus.cc:2932] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:08.833801 12992 raft_consensus.cc:2932] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:08.838104 13121 raft_consensus.cc:2932] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:08.841567 12587 catalog_manager.cc:5484] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:08.883703 22940 ts_tablet_manager.cc:902] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:19:08.889923 12860 consensus_queue.cc:237] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:08.892704 15753 raft_consensus.cc:1261] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:08.892868 12992 raft_consensus.cc:1261] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:08.893213 12796 consensus_peers.cc:485] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a -> Peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Couldn't send request to peer db7893bb29a047b3b82cc60bfd8930a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6546b77d0fd3434487dfe3abd335803e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:08.893628 22923 consensus_queue.cc:1035] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:08.893903 22910 consensus_queue.cc:1035] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:08.894212 22940 tablet_copy_client.cc:308] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.7:38851
I20240502 08:19:08.895290 15775 tablet_copy_service.cc:140] P 9333cb10227b4801b4fa7c24e22e131b: Received BeginTabletCopySession request for tablet 040878c4c4ae4684b02296ee9eac8985 from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:37259)
I20240502 08:19:08.895421 15775 tablet_copy_service.cc:161] P 9333cb10227b4801b4fa7c24e22e131b: Beginning new tablet copy session on tablet 040878c4c4ae4684b02296ee9eac8985 from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:37259: session id = e1dd14f7ca764a1490df419f13f2804a-040878c4c4ae4684b02296ee9eac8985
I20240502 08:19:08.900985 15775 tablet_copy_source_session.cc:205] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:08.902489 22940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 040878c4c4ae4684b02296ee9eac8985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:08.909662 22940 tablet_copy_client.cc:783] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:08.909855 22940 tablet_copy_client.cc:647] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:08.909713 22910 raft_consensus.cc:2932] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:08.910450 15753 raft_consensus.cc:2932] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:08.911657 22940 tablet_copy_client.cc:523] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:08.917846 12587 catalog_manager.cc:5484] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index -1 to 2, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:08.918530 22940 tablet_bootstrap.cc:492] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:19:08.918479 12992 raft_consensus.cc:2932] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:08.924703 22940 tablet_bootstrap.cc:492] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:08.924978 22940 tablet_bootstrap.cc:492] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:19:08.925166 22940 ts_tablet_manager.cc:1367] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:19:08.925653 22940 raft_consensus.cc:348] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:08.925799 22940 raft_consensus.cc:729] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:19:08.926017 22940 consensus_queue.cc:260] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:08.926944 22940 ts_tablet_manager.cc:1398] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:19:08.927675 15775 tablet_copy_service.cc:342] P 9333cb10227b4801b4fa7c24e22e131b: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-040878c4c4ae4684b02296ee9eac8985 received from {username='slave'} at 127.6.5.2:37259
I20240502 08:19:08.927781 15775 tablet_copy_service.cc:433] P 9333cb10227b4801b4fa7c24e22e131b: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-040878c4c4ae4684b02296ee9eac8985 on tablet 040878c4c4ae4684b02296ee9eac8985 with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:08.957851 12861 consensus_queue.cc:237] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:08.960044 13121 raft_consensus.cc:1261] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:08.960400 12795 consensus_peers.cc:485] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a -> Peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Couldn't send request to peer 9333cb10227b4801b4fa7c24e22e131b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c631a6dbc0d44f76abf1cf5a14ea4a6d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:08.960551 12727 raft_consensus.cc:1261] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:08.960671 22926 consensus_queue.cc:1035] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:08.961339 22926 consensus_queue.cc:1035] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:08.967657 22923 raft_consensus.cc:2932] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:08.968644 13121 raft_consensus.cc:2932] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:08.973609 12730 raft_consensus.cc:2932] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:08.973889 12587 catalog_manager.cc:5484] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:09.016630 15487 consensus_queue.cc:237] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:09.018824 15753 raft_consensus.cc:1261] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:09.019011 15622 raft_consensus.cc:1261] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:09.019296 22285 consensus_queue.cc:1035] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:09.019553 22447 consensus_queue.cc:1035] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:19:09.019801 15419 consensus_peers.cc:485] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f705e37c6be455eaa2a19145474bf9e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:09.023732 22285 raft_consensus.cc:2932] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:09.026640 15621 raft_consensus.cc:2932] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:09.029157 12588 catalog_manager.cc:5484] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:09.035208 15753 raft_consensus.cc:2932] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:09.083986 23004 ts_tablet_manager.cc:902] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:09.084718 12730 consensus_queue.cc:237] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:09.085316 23004 tablet_copy_client.cc:308] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:19:09.085896 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet 1f07b2caf1754c469a8e33382f615019 from peer db7893bb29a047b3b82cc60bfd8930a3 ({username='slave'} at 127.6.5.5:46815)
I20240502 08:19:09.086015 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet 1f07b2caf1754c469a8e33382f615019 from peer db7893bb29a047b3b82cc60bfd8930a3 at {username='slave'} at 127.6.5.5:46815: session id = db7893bb29a047b3b82cc60bfd8930a3-1f07b2caf1754c469a8e33382f615019
I20240502 08:19:09.087262 13121 raft_consensus.cc:1261] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:09.087761 22200 consensus_queue.cc:1035] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:09.088665 12751 tablet_copy_source_session.cc:205] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:09.089210 12992 raft_consensus.cc:1261] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:09.089695 22200 consensus_queue.cc:1035] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:09.090346 23011 ts_tablet_manager.cc:902] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:09.091230 23004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f07b2caf1754c469a8e33382f615019. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:09.092414 22745 raft_consensus.cc:2932] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:09.095041 12992 raft_consensus.cc:2932] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
W20240502 08:19:09.095386 12665 consensus_peers.cc:485] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee -> Peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Couldn't send request to peer 10f0e36203914b85b77e78609ffced98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c3f0742355442af8a29b971921b7d45. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:09.098302 13121 raft_consensus.cc:2932] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:09.099421 23004 tablet_copy_client.cc:783] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:09.099686 23004 tablet_copy_client.cc:647] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:09.100291 12587 catalog_manager.cc:5484] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:09.101419 23004 tablet_copy_client.cc:523] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:09.105499 23004 tablet_bootstrap.cc:492] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:19:09.125963 22940 ts_tablet_manager.cc:902] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:19:09.141106 23011 tablet_copy_client.cc:308] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:19:09.141716 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet 660b389a179f49eeb3007dea6ce078e9 from peer 10f0e36203914b85b77e78609ffced98 ({username='slave'} at 127.6.5.8:33311)
I20240502 08:19:09.141827 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet 660b389a179f49eeb3007dea6ce078e9 from peer 10f0e36203914b85b77e78609ffced98 at {username='slave'} at 127.6.5.8:33311: session id = 10f0e36203914b85b77e78609ffced98-660b389a179f49eeb3007dea6ce078e9
I20240502 08:19:09.144897 13012 tablet_copy_source_session.cc:205] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:09.146123 23011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660b389a179f49eeb3007dea6ce078e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:09.156209 23011 tablet_copy_client.cc:783] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:09.156455 23011 tablet_copy_client.cc:647] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:09.158363 23011 tablet_copy_client.cc:523] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:09.161921 23011 tablet_bootstrap.cc:492] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:19:09.167557 22940 tablet_copy_client.cc:308] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:19:09.168313 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet 087809a059d14b1f9a24ec3e956e1305 from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:36987)
I20240502 08:19:09.168445 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet 087809a059d14b1f9a24ec3e956e1305 from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:36987: session id = e1dd14f7ca764a1490df419f13f2804a-087809a059d14b1f9a24ec3e956e1305
I20240502 08:19:09.171021 13142 tablet_copy_source_session.cc:205] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:09.172210 22940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087809a059d14b1f9a24ec3e956e1305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:09.179018 22940 tablet_copy_client.cc:783] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:09.179217 22940 tablet_copy_client.cc:647] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:09.181126 23004 tablet_bootstrap.cc:492] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:09.181520 23004 tablet_bootstrap.cc:492] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap complete.
I20240502 08:19:09.181785 22940 tablet_copy_client.cc:523] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:09.181810 23004 ts_tablet_manager.cc:1367] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.076s user 0.057s sys 0.004s
I20240502 08:19:09.182653 23004 raft_consensus.cc:348] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:09.182926 23004 raft_consensus.cc:729] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:19:09.183195 23004 consensus_queue.cc:260] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:09.183861 23004 ts_tablet_manager.cc:1398] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:19:09.184602 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session db7893bb29a047b3b82cc60bfd8930a3-1f07b2caf1754c469a8e33382f615019 received from {username='slave'} at 127.6.5.5:46815
I20240502 08:19:09.184702 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session db7893bb29a047b3b82cc60bfd8930a3-1f07b2caf1754c469a8e33382f615019 on tablet 1f07b2caf1754c469a8e33382f615019 with peer db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:19:09.186357 22940 tablet_bootstrap.cc:492] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:19:09.197806 22940 tablet_bootstrap.cc:492] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:09.198141 22940 tablet_bootstrap.cc:492] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:19:09.198328 22940 ts_tablet_manager.cc:1367] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.002s
I20240502 08:19:09.198884 22940 raft_consensus.cc:348] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:09.199039 22940 raft_consensus.cc:729] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:19:09.199258 22940 consensus_queue.cc:260] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:09.199890 22940 ts_tablet_manager.cc:1398] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:19:09.200476 13142 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-087809a059d14b1f9a24ec3e956e1305 received from {username='slave'} at 127.6.5.2:36987
I20240502 08:19:09.200569 13142 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-087809a059d14b1f9a24ec3e956e1305 on tablet 087809a059d14b1f9a24ec3e956e1305 with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:09.212553 23011 tablet_bootstrap.cc:492] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:09.212911 23011 tablet_bootstrap.cc:492] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98: Bootstrap complete.
I20240502 08:19:09.213141 23011 ts_tablet_manager.cc:1367] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.051s user 0.036s sys 0.006s
I20240502 08:19:09.213779 23011 raft_consensus.cc:348] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:09.213982 23011 raft_consensus.cc:729] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:19:09.214185 23011 consensus_queue.cc:260] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:09.214910 23011 ts_tablet_manager.cc:1398] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 08:19:09.215481 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session 10f0e36203914b85b77e78609ffced98-660b389a179f49eeb3007dea6ce078e9 received from {username='slave'} at 127.6.5.8:33311
I20240502 08:19:09.215586 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session 10f0e36203914b85b77e78609ffced98-660b389a179f49eeb3007dea6ce078e9 on tablet 660b389a179f49eeb3007dea6ce078e9 with peer 10f0e36203914b85b77e78609ffced98
I20240502 08:19:09.229435 23040 ts_tablet_manager.cc:902] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:09.236506 13121 consensus_queue.cc:237] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:09.251758 23040 tablet_copy_client.cc:308] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:19:09.252614 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet 5be106986ad247c597666d46a1dede7d from peer 9333cb10227b4801b4fa7c24e22e131b ({username='slave'} at 127.6.5.7:55913)
I20240502 08:19:09.252749 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet 5be106986ad247c597666d46a1dede7d from peer 9333cb10227b4801b4fa7c24e22e131b at {username='slave'} at 127.6.5.7:55913: session id = 9333cb10227b4801b4fa7c24e22e131b-5be106986ad247c597666d46a1dede7d
I20240502 08:19:09.253279 12730 raft_consensus.cc:1261] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:19:09.253260 13056 consensus_peers.cc:485] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Couldn't send request to peer db7893bb29a047b3b82cc60bfd8930a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6cd36cfde5e480cad86a046d59e8940. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:09.253772 12992 raft_consensus.cc:1261] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:09.254235 22902 consensus_queue.cc:1035] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:09.255182 22902 consensus_queue.cc:1035] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:09.255318 13012 tablet_copy_source_session.cc:205] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:09.258675 23040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be106986ad247c597666d46a1dede7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:09.259145 12860 raft_consensus.cc:1203] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240502 08:19:09.259147 22651 raft_consensus.cc:2932] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:09.260080 12992 raft_consensus.cc:2932] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:09.265023 12730 raft_consensus.cc:2932] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:09.267385 12588 catalog_manager.cc:5484] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:09.274284 23040 tablet_copy_client.cc:783] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:09.274550 23040 tablet_copy_client.cc:647] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:09.276945 23040 tablet_copy_client.cc:523] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:09.282511 23040 tablet_bootstrap.cc:492] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:19:09.288592 23040 tablet_bootstrap.cc:492] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:09.289011 23040 tablet_bootstrap.cc:492] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap complete.
I20240502 08:19:09.289273 23040 ts_tablet_manager.cc:1367] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20240502 08:19:09.289963 23040 raft_consensus.cc:348] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:09.290781 23040 raft_consensus.cc:729] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:19:09.290978 23040 consensus_queue.cc:260] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:09.291716 23040 ts_tablet_manager.cc:1398] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:19:09.293352 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session 9333cb10227b4801b4fa7c24e22e131b-5be106986ad247c597666d46a1dede7d received from {username='slave'} at 127.6.5.7:55913
I20240502 08:19:09.293463 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session 9333cb10227b4801b4fa7c24e22e131b-5be106986ad247c597666d46a1dede7d on tablet 5be106986ad247c597666d46a1dede7d with peer 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:19:09.308670 13121 consensus_queue.cc:237] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:09.311090 15621 raft_consensus.cc:1261] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:09.311106 13056 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Couldn't send request to peer 29b035434087435cbaf4ebcac98eedee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd0a5745bdba436e848ba33029ef01b4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:09.311103 15753 raft_consensus.cc:1261] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:09.311630 22651 consensus_queue.cc:1035] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:09.311832 22518 consensus_queue.cc:1035] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:09.314532 22651 raft_consensus.cc:2932] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:09.317795 15621 raft_consensus.cc:2932] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:09.319377 15753 raft_consensus.cc:2932] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:09.320724 12587 catalog_manager.cc:5484] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:09.321835 23011 ts_tablet_manager.cc:902] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98: Initiating tablet copy from peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:19:09.345081 23011 tablet_copy_client.cc:308] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:36933
I20240502 08:19:09.345839 12882 tablet_copy_service.cc:140] P e1dd14f7ca764a1490df419f13f2804a: Received BeginTabletCopySession request for tablet 581d2ed76e2e446eb84b94231ebb97f4 from peer 10f0e36203914b85b77e78609ffced98 ({username='slave'} at 127.6.5.8:37245)
I20240502 08:19:09.345958 12882 tablet_copy_service.cc:161] P e1dd14f7ca764a1490df419f13f2804a: Beginning new tablet copy session on tablet 581d2ed76e2e446eb84b94231ebb97f4 from peer 10f0e36203914b85b77e78609ffced98 at {username='slave'} at 127.6.5.8:37245: session id = 10f0e36203914b85b77e78609ffced98-581d2ed76e2e446eb84b94231ebb97f4
I20240502 08:19:09.348343 12882 tablet_copy_source_session.cc:205] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:09.349620 23011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 581d2ed76e2e446eb84b94231ebb97f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:09.355952 23011 tablet_copy_client.cc:783] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:09.356146 23011 tablet_copy_client.cc:647] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:09.359412 23011 tablet_copy_client.cc:523] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:09.364933 23011 tablet_bootstrap.cc:492] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:19:09.389803 12992 consensus_queue.cc:237] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:09.392304 15486 raft_consensus.cc:1261] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:09.392495 15621 raft_consensus.cc:1261] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:09.392915 23090 consensus_queue.cc:1035] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:09.393187 22845 consensus_queue.cc:1035] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:19:09.393728 12926 consensus_peers.cc:485] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82ecdc518ec94e43b0867a0786575304. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:09.404237 23090 raft_consensus.cc:2932] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:09.408571 15621 raft_consensus.cc:2932] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:09.407191 15486 raft_consensus.cc:2932] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:09.410754 23011 tablet_bootstrap.cc:492] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:09.411090 23011 tablet_bootstrap.cc:492] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98: Bootstrap complete.
I20240502 08:19:09.411379 23011 ts_tablet_manager.cc:1367] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.047s user 0.043s sys 0.000s
I20240502 08:19:09.411597 12587 catalog_manager.cc:5484] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:09.412194 23011 raft_consensus.cc:348] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:09.412395 23011 raft_consensus.cc:729] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:19:09.412606 23011 consensus_queue.cc:260] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:09.413311 23011 ts_tablet_manager.cc:1398] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:19:09.414758 12882 tablet_copy_service.cc:342] P e1dd14f7ca764a1490df419f13f2804a: Request end of tablet copy session 10f0e36203914b85b77e78609ffced98-581d2ed76e2e446eb84b94231ebb97f4 received from {username='slave'} at 127.6.5.8:37245
I20240502 08:19:09.414877 12882 tablet_copy_service.cc:433] P e1dd14f7ca764a1490df419f13f2804a: ending tablet copy session 10f0e36203914b85b77e78609ffced98-581d2ed76e2e446eb84b94231ebb97f4 on tablet 581d2ed76e2e446eb84b94231ebb97f4 with peer 10f0e36203914b85b77e78609ffced98
I20240502 08:19:09.479221 15887 raft_consensus.cc:1203] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:09.501325 12730 consensus_queue.cc:237] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
W20240502 08:19:09.503523 12665 consensus_peers.cc:485] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee -> Peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Couldn't send request to peer 10f0e36203914b85b77e78609ffced98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fba813ac35c4382a0917604c2057496. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:09.503587 13121 raft_consensus.cc:1261] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:09.503625 12861 raft_consensus.cc:1261] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:09.504366 22200 consensus_queue.cc:1035] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:09.505270 22745 consensus_queue.cc:1035] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:09.515502 22745 raft_consensus.cc:2932] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:09.516130 13121 raft_consensus.cc:2932] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:09.519351 23004 ts_tablet_manager.cc:902] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3: Initiating tablet copy from peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:19:09.520520 23040 ts_tablet_manager.cc:902] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b: Initiating tablet copy from peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:19:09.524019 12587 catalog_manager.cc:5484] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:09.524612 12860 raft_consensus.cc:2932] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:09.542762 23004 tablet_copy_client.cc:308] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:36933
I20240502 08:19:09.543330 12882 tablet_copy_service.cc:140] P e1dd14f7ca764a1490df419f13f2804a: Received BeginTabletCopySession request for tablet 6546b77d0fd3434487dfe3abd335803e from peer db7893bb29a047b3b82cc60bfd8930a3 ({username='slave'} at 127.6.5.5:45483)
I20240502 08:19:09.543447 12882 tablet_copy_service.cc:161] P e1dd14f7ca764a1490df419f13f2804a: Beginning new tablet copy session on tablet 6546b77d0fd3434487dfe3abd335803e from peer db7893bb29a047b3b82cc60bfd8930a3 at {username='slave'} at 127.6.5.5:45483: session id = db7893bb29a047b3b82cc60bfd8930a3-6546b77d0fd3434487dfe3abd335803e
I20240502 08:19:09.545843 12882 tablet_copy_source_session.cc:205] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:09.547220 23040 tablet_copy_client.cc:308] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:36933
I20240502 08:19:09.547310 23004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6546b77d0fd3434487dfe3abd335803e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:09.547758 12882 tablet_copy_service.cc:140] P e1dd14f7ca764a1490df419f13f2804a: Received BeginTabletCopySession request for tablet c631a6dbc0d44f76abf1cf5a14ea4a6d from peer 9333cb10227b4801b4fa7c24e22e131b ({username='slave'} at 127.6.5.7:57025)
I20240502 08:19:09.547856 12882 tablet_copy_service.cc:161] P e1dd14f7ca764a1490df419f13f2804a: Beginning new tablet copy session on tablet c631a6dbc0d44f76abf1cf5a14ea4a6d from peer 9333cb10227b4801b4fa7c24e22e131b at {username='slave'} at 127.6.5.7:57025: session id = 9333cb10227b4801b4fa7c24e22e131b-c631a6dbc0d44f76abf1cf5a14ea4a6d
I20240502 08:19:09.549736 12882 tablet_copy_source_session.cc:205] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:09.550794 23040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c631a6dbc0d44f76abf1cf5a14ea4a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:09.552332 23004 tablet_copy_client.cc:783] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:09.552560 23004 tablet_copy_client.cc:647] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:09.553922 23004 tablet_copy_client.cc:523] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:09.556545 15486 raft_consensus.cc:1203] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:09.557194 23040 tablet_copy_client.cc:783] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:09.557379 23040 tablet_copy_client.cc:647] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:09.558638 23040 tablet_copy_client.cc:523] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:09.560972 23004 tablet_bootstrap.cc:492] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:19:09.561339 23040 tablet_bootstrap.cc:492] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:19:09.569792 23004 tablet_bootstrap.cc:492] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:09.570384 23004 tablet_bootstrap.cc:492] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap complete.
I20240502 08:19:09.570631 23004 ts_tablet_manager.cc:1367] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240502 08:19:09.571301 23004 raft_consensus.cc:348] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:09.571545 23004 raft_consensus.cc:729] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:19:09.571779 23004 consensus_queue.cc:260] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:09.572283 23040 tablet_bootstrap.cc:492] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:09.572559 23040 tablet_bootstrap.cc:492] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap complete.
I20240502 08:19:09.572767 23040 ts_tablet_manager.cc:1367] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.003s
I20240502 08:19:09.573133 23004 ts_tablet_manager.cc:1398] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:19:09.573336 23040 raft_consensus.cc:348] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:09.573529 23040 raft_consensus.cc:729] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:19:09.573657 12882 tablet_copy_service.cc:342] P e1dd14f7ca764a1490df419f13f2804a: Request end of tablet copy session db7893bb29a047b3b82cc60bfd8930a3-6546b77d0fd3434487dfe3abd335803e received from {username='slave'} at 127.6.5.5:45483
I20240502 08:19:09.573756 12882 tablet_copy_service.cc:433] P e1dd14f7ca764a1490df419f13f2804a: ending tablet copy session db7893bb29a047b3b82cc60bfd8930a3-6546b77d0fd3434487dfe3abd335803e on tablet 6546b77d0fd3434487dfe3abd335803e with peer db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:19:09.573740 23040 consensus_queue.cc:260] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:09.574220 23040 ts_tablet_manager.cc:1398] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:19:09.574942 12882 tablet_copy_service.cc:342] P e1dd14f7ca764a1490df419f13f2804a: Request end of tablet copy session 9333cb10227b4801b4fa7c24e22e131b-c631a6dbc0d44f76abf1cf5a14ea4a6d received from {username='slave'} at 127.6.5.7:57025
I20240502 08:19:09.575028 12882 tablet_copy_service.cc:433] P e1dd14f7ca764a1490df419f13f2804a: ending tablet copy session 9333cb10227b4801b4fa7c24e22e131b-c631a6dbc0d44f76abf1cf5a14ea4a6d on tablet c631a6dbc0d44f76abf1cf5a14ea4a6d with peer 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:19:09.603762 12730 consensus_queue.cc:237] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
W20240502 08:19:09.606150 12663 consensus_peers.cc:485] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee -> Peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Couldn't send request to peer 9333cb10227b4801b4fa7c24e22e131b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e76ea85438940399e164313caddf121. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:09.606730 15621 raft_consensus.cc:1261] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:09.607069 15486 raft_consensus.cc:1261] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:09.607190 22745 consensus_queue.cc:1035] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:09.607846 22745 consensus_queue.cc:1035] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:09.610998 22200 raft_consensus.cc:2932] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:09.611650 15486 raft_consensus.cc:2932] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:09.615310 15622 raft_consensus.cc:2932] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:09.620474 12588 catalog_manager.cc:5484] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:09.633559 23011 ts_tablet_manager.cc:902] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:09.637797 22940 ts_tablet_manager.cc:902] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:19:09.652340 15755 raft_consensus.cc:1203] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:09.661310 22940 tablet_copy_client.cc:308] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:19:09.662061 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet 5f705e37c6be455eaa2a19145474bf9e from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:60127)
I20240502 08:19:09.662220 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet 5f705e37c6be455eaa2a19145474bf9e from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:60127: session id = e1dd14f7ca764a1490df419f13f2804a-5f705e37c6be455eaa2a19145474bf9e
I20240502 08:19:09.665120 15508 tablet_copy_source_session.cc:205] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:09.670667 22940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f705e37c6be455eaa2a19145474bf9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:09.674846 23011 tablet_copy_client.cc:308] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:19:09.675534 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet 5c3f0742355442af8a29b971921b7d45 from peer 10f0e36203914b85b77e78609ffced98 ({username='slave'} at 127.6.5.8:39069)
I20240502 08:19:09.675663 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet 5c3f0742355442af8a29b971921b7d45 from peer 10f0e36203914b85b77e78609ffced98 at {username='slave'} at 127.6.5.8:39069: session id = 10f0e36203914b85b77e78609ffced98-5c3f0742355442af8a29b971921b7d45
I20240502 08:19:09.678292 12751 tablet_copy_source_session.cc:205] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:09.679471 23011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c3f0742355442af8a29b971921b7d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:09.685846 23011 tablet_copy_client.cc:783] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:09.686098 23011 tablet_copy_client.cc:647] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:09.686722 22940 tablet_copy_client.cc:783] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:09.686937 22940 tablet_copy_client.cc:647] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:09.687800 23011 tablet_copy_client.cc:523] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:09.690690 22940 tablet_copy_client.cc:523] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:09.691231 23011 tablet_bootstrap.cc:492] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:19:09.696158 22940 tablet_bootstrap.cc:492] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:19:09.703372 23004 ts_tablet_manager.cc:902] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:19:09.711338 12860 raft_consensus.cc:1203] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:09.729564 22940 tablet_bootstrap.cc:492] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:09.729920 22940 tablet_bootstrap.cc:492] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:19:09.730150 22940 ts_tablet_manager.cc:1367] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.004s
I20240502 08:19:09.733765 15887 raft_consensus.cc:1203] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:09.737051 22940 raft_consensus.cc:348] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:09.737293 22940 raft_consensus.cc:729] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:19:09.737521 22940 consensus_queue.cc:260] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:09.744307 22940 ts_tablet_manager.cc:1398] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240502 08:19:09.746487 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-5f705e37c6be455eaa2a19145474bf9e received from {username='slave'} at 127.6.5.2:60127
I20240502 08:19:09.746615 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-5f705e37c6be455eaa2a19145474bf9e on tablet 5f705e37c6be455eaa2a19145474bf9e with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:09.781380 15887 consensus_queue.cc:237] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:09.785851 23011 tablet_bootstrap.cc:492] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:09.786279 23011 tablet_bootstrap.cc:492] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98: Bootstrap complete.
I20240502 08:19:09.786504 23004 tablet_copy_client.cc:308] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:19:09.786685 23011 ts_tablet_manager.cc:1367] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.096s user 0.046s sys 0.019s
I20240502 08:19:09.787096 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet c6cd36cfde5e480cad86a046d59e8940 from peer db7893bb29a047b3b82cc60bfd8930a3 ({username='slave'} at 127.6.5.5:59601)
I20240502 08:19:09.787276 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet c6cd36cfde5e480cad86a046d59e8940 from peer db7893bb29a047b3b82cc60bfd8930a3 at {username='slave'} at 127.6.5.5:59601: session id = db7893bb29a047b3b82cc60bfd8930a3-c6cd36cfde5e480cad86a046d59e8940
W20240502 08:19:09.788491 15818 consensus_peers.cc:485] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 -> Peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Couldn't send request to peer 6a0b05f3250d415590b0edf6b44fbe1f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d57fc923fe946afa65a6aa8ce52ba7b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:09.788547 12730 raft_consensus.cc:1261] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:09.788655 12860 raft_consensus.cc:1261] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:09.788973 21715 consensus_queue.cc:1035] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:09.789184 22559 consensus_queue.cc:1035] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:09.789927 13142 tablet_copy_source_session.cc:205] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:09.792771 23004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6cd36cfde5e480cad86a046d59e8940. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:09.789743 23011 raft_consensus.cc:348] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:09.796981 23011 raft_consensus.cc:729] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:19:09.797189 23011 consensus_queue.cc:260] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:09.800549 23011 ts_tablet_manager.cc:1398] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240502 08:19:09.801131 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 10f0e36203914b85b77e78609ffced98-5c3f0742355442af8a29b971921b7d45 received from {username='slave'} at 127.6.5.8:39069
I20240502 08:19:09.801232 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 10f0e36203914b85b77e78609ffced98-5c3f0742355442af8a29b971921b7d45 on tablet 5c3f0742355442af8a29b971921b7d45 with peer 10f0e36203914b85b77e78609ffced98
I20240502 08:19:09.801163 23004 tablet_copy_client.cc:783] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:09.803119 23004 tablet_copy_client.cc:647] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:09.805569 23004 tablet_copy_client.cc:523] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:09.808634 23004 tablet_bootstrap.cc:492] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:19:09.823470 23161 raft_consensus.cc:2932] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:09.827121 12861 raft_consensus.cc:2932] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:09.833922 12728 raft_consensus.cc:2932] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:09.841311 12587 catalog_manager.cc:5484] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:09.871794 23004 tablet_bootstrap.cc:492] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:09.872128 23004 tablet_bootstrap.cc:492] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap complete.
I20240502 08:19:09.872365 23004 ts_tablet_manager.cc:1367] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.064s user 0.047s sys 0.002s
I20240502 08:19:09.872949 23004 raft_consensus.cc:348] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:09.873158 23004 raft_consensus.cc:729] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:19:09.873356 23004 consensus_queue.cc:260] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:09.873905 23004 ts_tablet_manager.cc:1398] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 08:19:09.874679 13142 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session db7893bb29a047b3b82cc60bfd8930a3-c6cd36cfde5e480cad86a046d59e8940 received from {username='slave'} at 127.6.5.5:59601
I20240502 08:19:09.874796 13142 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session db7893bb29a047b3b82cc60bfd8930a3-c6cd36cfde5e480cad86a046d59e8940 on tablet c6cd36cfde5e480cad86a046d59e8940 with peer db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:19:09.901041 22940 ts_tablet_manager.cc:902] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:09.913003 22940 tablet_copy_client.cc:308] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:19:09.913563 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet 82ecdc518ec94e43b0867a0786575304 from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:47079)
I20240502 08:19:09.913705 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet 82ecdc518ec94e43b0867a0786575304 from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:47079: session id = e1dd14f7ca764a1490df419f13f2804a-82ecdc518ec94e43b0867a0786575304
I20240502 08:19:09.916568 13012 tablet_copy_source_session.cc:205] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:09.917743 22940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ecdc518ec94e43b0867a0786575304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:09.935648 23198 ts_tablet_manager.cc:902] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:19:09.940306 22940 tablet_copy_client.cc:783] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:09.940532 22940 tablet_copy_client.cc:647] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:09.941964 22940 tablet_copy_client.cc:523] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:09.944780 23198 tablet_copy_client.cc:308] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:19:09.945109 22940 tablet_bootstrap.cc:492] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:19:09.945488 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet fd0a5745bdba436e848ba33029ef01b4 from peer 29b035434087435cbaf4ebcac98eedee ({username='slave'} at 127.6.5.1:46969)
I20240502 08:19:09.945641 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet fd0a5745bdba436e848ba33029ef01b4 from peer 29b035434087435cbaf4ebcac98eedee at {username='slave'} at 127.6.5.1:46969: session id = 29b035434087435cbaf4ebcac98eedee-fd0a5745bdba436e848ba33029ef01b4
I20240502 08:19:09.948437 13142 tablet_copy_source_session.cc:205] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:09.949991 22940 tablet_bootstrap.cc:492] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:09.950024 23198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0a5745bdba436e848ba33029ef01b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:09.950265 22940 tablet_bootstrap.cc:492] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:19:09.950507 22940 ts_tablet_manager.cc:1367] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.005s user 0.008s sys 0.000s
I20240502 08:19:09.951078 22940 raft_consensus.cc:348] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:09.951267 22940 raft_consensus.cc:729] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:19:09.951469 22940 consensus_queue.cc:260] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:09.957338 22940 ts_tablet_manager.cc:1398] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:19:09.958050 23198 tablet_copy_client.cc:783] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:09.958256 23198 tablet_copy_client.cc:647] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:09.959434 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-82ecdc518ec94e43b0867a0786575304 received from {username='slave'} at 127.6.5.2:47079
I20240502 08:19:09.959527 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-82ecdc518ec94e43b0867a0786575304 on tablet 82ecdc518ec94e43b0867a0786575304 with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:09.959627 23198 tablet_copy_client.cc:523] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:09.963409 23198 tablet_bootstrap.cc:492] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:19:09.977975 23198 tablet_bootstrap.cc:492] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:09.978309 23198 tablet_bootstrap.cc:492] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee: Bootstrap complete.
I20240502 08:19:09.978547 23198 ts_tablet_manager.cc:1367] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.006s
I20240502 08:19:09.979148 23198 raft_consensus.cc:348] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:09.979351 23198 raft_consensus.cc:729] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:19:09.979560 23198 consensus_queue.cc:260] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:09.980086 23198 ts_tablet_manager.cc:1398] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:19:09.985198 13142 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session 29b035434087435cbaf4ebcac98eedee-fd0a5745bdba436e848ba33029ef01b4 received from {username='slave'} at 127.6.5.1:46969
I20240502 08:19:09.985304 13142 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session 29b035434087435cbaf4ebcac98eedee-fd0a5745bdba436e848ba33029ef01b4 on tablet fd0a5745bdba436e848ba33029ef01b4 with peer 29b035434087435cbaf4ebcac98eedee
I20240502 08:19:10.005919 15486 raft_consensus.cc:1203] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:10.020223 23040 ts_tablet_manager.cc:902] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:10.035207 15755 raft_consensus.cc:1203] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:10.040357 23011 ts_tablet_manager.cc:902] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:10.102634 23040 tablet_copy_client.cc:308] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:19:10.103150 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet 2e76ea85438940399e164313caddf121 from peer 9333cb10227b4801b4fa7c24e22e131b ({username='slave'} at 127.6.5.7:35555)
I20240502 08:19:10.103298 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet 2e76ea85438940399e164313caddf121 from peer 9333cb10227b4801b4fa7c24e22e131b at {username='slave'} at 127.6.5.7:35555: session id = 9333cb10227b4801b4fa7c24e22e131b-2e76ea85438940399e164313caddf121
I20240502 08:19:10.106050 12751 tablet_copy_source_session.cc:205] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:10.107234 23040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e76ea85438940399e164313caddf121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:10.110754 23011 tablet_copy_client.cc:308] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:19:10.111387 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet 1fba813ac35c4382a0917604c2057496 from peer 10f0e36203914b85b77e78609ffced98 ({username='slave'} at 127.6.5.8:39069)
I20240502 08:19:10.111514 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet 1fba813ac35c4382a0917604c2057496 from peer 10f0e36203914b85b77e78609ffced98 at {username='slave'} at 127.6.5.8:39069: session id = 10f0e36203914b85b77e78609ffced98-1fba813ac35c4382a0917604c2057496
I20240502 08:19:10.112464 23040 tablet_copy_client.cc:783] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:10.112655 23040 tablet_copy_client.cc:647] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:10.113795 12751 tablet_copy_source_session.cc:205] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:10.115497 23011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fba813ac35c4382a0917604c2057496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:10.123778 23011 tablet_copy_client.cc:783] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:10.123996 23011 tablet_copy_client.cc:647] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:10.126859 23040 tablet_copy_client.cc:523] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:10.129940 23040 tablet_bootstrap.cc:492] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:19:10.137722 12860 raft_consensus.cc:1203] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:10.152179 23011 tablet_copy_client.cc:523] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:10.158207 23011 tablet_bootstrap.cc:492] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:19:10.162982 23040 tablet_bootstrap.cc:492] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:10.163251 23040 tablet_bootstrap.cc:492] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap complete.
I20240502 08:19:10.163439 23040 ts_tablet_manager.cc:1367] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.014s
I20240502 08:19:10.163988 23040 raft_consensus.cc:348] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:10.164156 23040 raft_consensus.cc:729] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:19:10.164340 23040 consensus_queue.cc:260] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:10.164942 23040 ts_tablet_manager.cc:1398] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:19:10.165810 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 9333cb10227b4801b4fa7c24e22e131b-2e76ea85438940399e164313caddf121 received from {username='slave'} at 127.6.5.7:35555
I20240502 08:19:10.165920 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 9333cb10227b4801b4fa7c24e22e131b-2e76ea85438940399e164313caddf121 on tablet 2e76ea85438940399e164313caddf121 with peer 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:19:10.186376 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:10.207129 23011 tablet_bootstrap.cc:492] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:10.207577 23011 tablet_bootstrap.cc:492] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98: Bootstrap complete.
I20240502 08:19:10.207866 23011 ts_tablet_manager.cc:1367] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.052s user 0.041s sys 0.002s
I20240502 08:19:10.208580 23011 raft_consensus.cc:348] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:10.208794 23011 raft_consensus.cc:729] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:19:10.209003 23011 consensus_queue.cc:260] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:10.212232 15887 raft_consensus.cc:1203] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:10.213516 23011 ts_tablet_manager.cc:1398] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240502 08:19:10.214177 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 10f0e36203914b85b77e78609ffced98-1fba813ac35c4382a0917604c2057496 received from {username='slave'} at 127.6.5.8:39069
I20240502 08:19:10.214288 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 10f0e36203914b85b77e78609ffced98-1fba813ac35c4382a0917604c2057496 on tablet 1fba813ac35c4382a0917604c2057496 with peer 10f0e36203914b85b77e78609ffced98
I20240502 08:19:10.249694 15486 raft_consensus.cc:1203] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:10.263978 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:10.293802 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:10.319535 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:10.336222 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:10.365092 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:10.365348 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:10.367972 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:10.374675 23255 ts_tablet_manager.cc:902] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f: Initiating tablet copy from peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:19:10.410094 15753 raft_consensus.cc:1203] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:10.480063 12860 raft_consensus.cc:1203] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:10.530675 23255 tablet_copy_client.cc:308] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.8:46303
I20240502 08:19:10.531203 15907 tablet_copy_service.cc:140] P 10f0e36203914b85b77e78609ffced98: Received BeginTabletCopySession request for tablet 0d57fc923fe946afa65a6aa8ce52ba7b from peer 6a0b05f3250d415590b0edf6b44fbe1f ({username='slave'} at 127.6.5.4:32771)
I20240502 08:19:10.531319 15907 tablet_copy_service.cc:161] P 10f0e36203914b85b77e78609ffced98: Beginning new tablet copy session on tablet 0d57fc923fe946afa65a6aa8ce52ba7b from peer 6a0b05f3250d415590b0edf6b44fbe1f at {username='slave'} at 127.6.5.4:32771: session id = 6a0b05f3250d415590b0edf6b44fbe1f-0d57fc923fe946afa65a6aa8ce52ba7b
I20240502 08:19:10.533766 15907 tablet_copy_source_session.cc:205] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:10.535336 12730 raft_consensus.cc:1203] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:10.551528 23255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d57fc923fe946afa65a6aa8ce52ba7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:10.558203 23255 tablet_copy_client.cc:783] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:10.560488 23255 tablet_copy_client.cc:647] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:10.562541 23255 tablet_copy_client.cc:523] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:10.565146 23255 tablet_bootstrap.cc:492] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:19:10.571353 23255 tablet_bootstrap.cc:492] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:10.572448 23255 tablet_bootstrap.cc:492] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap complete.
I20240502 08:19:10.572701 23255 ts_tablet_manager.cc:1367] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20240502 08:19:10.573220 23255 raft_consensus.cc:348] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:10.573395 23255 raft_consensus.cc:729] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:19:10.573621 23255 consensus_queue.cc:260] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:10.575227 23255 ts_tablet_manager.cc:1398] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:19:10.575911 15907 tablet_copy_service.cc:342] P 10f0e36203914b85b77e78609ffced98: Request end of tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-0d57fc923fe946afa65a6aa8ce52ba7b received from {username='slave'} at 127.6.5.4:32771
I20240502 08:19:10.576020 15907 tablet_copy_service.cc:433] P 10f0e36203914b85b77e78609ffced98: ending tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-0d57fc923fe946afa65a6aa8ce52ba7b on tablet 0d57fc923fe946afa65a6aa8ce52ba7b with peer 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:19:10.620316 15886 raft_consensus.cc:1203] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:10.862951 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:10.887380 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:10.895116 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:10.898171 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:10.941385 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:10.941690 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:10.961127 13121 raft_consensus.cc:1203] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:11.003922 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:11.026166 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:11.142534 12730 consensus_queue.cc:237] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:11.144290 15753 raft_consensus.cc:1261] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:11.144290 15487 raft_consensus.cc:1261] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:11.144742 22200 consensus_queue.cc:1035] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:11.144997 23015 consensus_queue.cc:1035] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:11.147711 23015 raft_consensus.cc:2932] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:11.148213 15753 raft_consensus.cc:2932] T 2e76ea85438940399e164313caddf121 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:11.148213 15487 raft_consensus.cc:2932] T 2e76ea85438940399e164313caddf121 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:11.154350 12579 catalog_manager.cc:5484] T 2e76ea85438940399e164313caddf121 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:11.178359 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e76ea85438940399e164313caddf121 with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:11.178884 23357 tablet_replica.cc:330] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:11.179066 23357 raft_consensus.cc:2229] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:11.179255 23357 raft_consensus.cc:2258] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:11.179872 23357 ts_tablet_manager.cc:1875] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:11.185263 23357 ts_tablet_manager.cc:1888] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:11.185392 23357 log.cc:1196] T 2e76ea85438940399e164313caddf121 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/2e76ea85438940399e164313caddf121
I20240502 08:19:11.186250 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 2e76ea85438940399e164313caddf121 (table rebalancer_extra_table_16 [id=ac73d0460aa5473a8d5bdc70b924fe3f]) successfully deleted
I20240502 08:19:11.407534 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:11.407922 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:11.444684 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:11.490615 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:11.501245 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:11.514210 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:11.559242 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:11.559600 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:11.751487 12992 consensus_queue.cc:237] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:11.753887 15487 raft_consensus.cc:1261] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:11.753916 12861 raft_consensus.cc:1261] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:11.754416 23090 consensus_queue.cc:1035] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:11.754657 22849 consensus_queue.cc:1035] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:11.757465 23090 raft_consensus.cc:2932] T 82ecdc518ec94e43b0867a0786575304 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:11.757884 12861 raft_consensus.cc:2932] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:11.764477 12581 catalog_manager.cc:5484] T 82ecdc518ec94e43b0867a0786575304 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:11.776719 15487 raft_consensus.cc:2932] T 82ecdc518ec94e43b0867a0786575304 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:11.792310 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ecdc518ec94e43b0867a0786575304 with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:11.798877 23433 tablet_replica.cc:330] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:11.799118 23433 raft_consensus.cc:2229] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:11.799327 23433 raft_consensus.cc:2258] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:11.800904 23433 ts_tablet_manager.cc:1875] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:11.809880 23433 ts_tablet_manager.cc:1888] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:11.809996 23433 log.cc:1196] T 82ecdc518ec94e43b0867a0786575304 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/82ecdc518ec94e43b0867a0786575304
I20240502 08:19:11.810570 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 82ecdc518ec94e43b0867a0786575304 (table rebalancer_extra_table_32 [id=9326d144342b43d1ab9279d9cca295a0]) successfully deleted
I20240502 08:19:12.027045 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:12.035647 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:12.056607 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:12.080000 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:12.080943 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:12.152109 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:12.196285 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:12.226524 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:12.328809 13122 consensus_queue.cc:237] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:12.330411 12728 raft_consensus.cc:1261] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:12.330613 15619 raft_consensus.cc:1261] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:12.330947 23311 consensus_queue.cc:1035] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:12.331521 23311 consensus_queue.cc:1035] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:12.336268 22902 raft_consensus.cc:2932] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:12.336827 15619 raft_consensus.cc:2932] T fd0a5745bdba436e848ba33029ef01b4 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:12.343823 12580 catalog_manager.cc:5484] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:12.362920 12730 raft_consensus.cc:2932] T fd0a5745bdba436e848ba33029ef01b4 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:12.372231 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0a5745bdba436e848ba33029ef01b4 with delete_type TABLET_DATA_TOMBSTONED (TS 9333cb10227b4801b4fa7c24e22e131b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45134
I20240502 08:19:12.380592 23514 tablet_replica.cc:330] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:19:12.380792 23514 raft_consensus.cc:2229] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:12.381016 23514 raft_consensus.cc:2258] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:12.381609 23514 ts_tablet_manager.cc:1875] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:12.393167 23514 ts_tablet_manager.cc:1888] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:12.393270 23514 log.cc:1196] T fd0a5745bdba436e848ba33029ef01b4 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/fd0a5745bdba436e848ba33029ef01b4
I20240502 08:19:12.393916 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet fd0a5745bdba436e848ba33029ef01b4 (table rebalancer_extra_table_42 [id=2f80953a600947b79f5e500d1cb36f09]) successfully deleted
I20240502 08:19:12.658908 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:12.677764 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:12.692873 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:12.734601 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:12.747220 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:12.761410 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:12.840046 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:12.899724 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 08:19:12.900642 13058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.6:44983, user_credentials={real_user=slave}} blocked reactor thread for 133209us
W20240502 08:19:12.930778 15688 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.1:33183, user_credentials={real_user=slave}} blocked reactor thread for 85779.2us
I20240502 08:19:13.005185 13121 consensus_queue.cc:237] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:13.007272 15486 raft_consensus.cc:1261] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:13.007350 12730 raft_consensus.cc:1261] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:13.007730 23573 consensus_queue.cc:1035] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:13.007944 23572 consensus_queue.cc:1035] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:13.010170 22902 raft_consensus.cc:2932] T c6cd36cfde5e480cad86a046d59e8940 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:13.010779 15486 raft_consensus.cc:2932] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:13.010766 12730 raft_consensus.cc:2932] T c6cd36cfde5e480cad86a046d59e8940 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:13.016440 12581 catalog_manager.cc:5484] T c6cd36cfde5e480cad86a046d59e8940 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:13.022395 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet c6cd36cfde5e480cad86a046d59e8940 with delete_type TABLET_DATA_TOMBSTONED (TS 60b6e265e7754b7187307d05ef405dd4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38740
I20240502 08:19:13.026216 23600 tablet_replica.cc:330] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:19:13.026386 23600 raft_consensus.cc:2229] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:13.026610 23600 raft_consensus.cc:2258] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:13.027179 23600 ts_tablet_manager.cc:1875] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:13.032575 23600 ts_tablet_manager.cc:1888] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:13.032713 23600 log.cc:1196] T c6cd36cfde5e480cad86a046d59e8940 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/c6cd36cfde5e480cad86a046d59e8940
I20240502 08:19:13.035622 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet c6cd36cfde5e480cad86a046d59e8940 (table rebalance_test_table_11 [id=02e24b5edb434940bb115fce3e1947fb]) successfully deleted
I20240502 08:19:13.269572 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:13.305009 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:13.304996 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:13.321559 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:13.332913 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:13.361222 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:13.391122 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:13.403049 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:13.565455 12730 consensus_queue.cc:237] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:13.567238 12991 raft_consensus.cc:1261] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:13.567242 15887 raft_consensus.cc:1261] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:13.567806 23015 consensus_queue.cc:1035] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:13.570250 23015 consensus_queue.cc:1035] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:13.571142 23015 raft_consensus.cc:2932] T 5c3f0742355442af8a29b971921b7d45 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:13.573487 12991 raft_consensus.cc:2932] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:13.575084 15887 raft_consensus.cc:2932] T 5c3f0742355442af8a29b971921b7d45 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:13.581449 12581 catalog_manager.cc:5484] T 5c3f0742355442af8a29b971921b7d45 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:13.589031 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c3f0742355442af8a29b971921b7d45 with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:19:13.592693 23683 tablet_replica.cc:330] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:19:13.592866 23683 raft_consensus.cc:2229] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:13.593097 23683 raft_consensus.cc:2258] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:13.594636 23683 ts_tablet_manager.cc:1875] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:13.604835 23683 ts_tablet_manager.cc:1888] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:13.604998 23683 log.cc:1196] T 5c3f0742355442af8a29b971921b7d45 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/5c3f0742355442af8a29b971921b7d45
I20240502 08:19:13.605700 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 5c3f0742355442af8a29b971921b7d45 (table rebalance_test_table_9 [id=ab2e0dad5c0341efa8c2321070416a08]) successfully deleted
I20240502 08:19:13.835494 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:13.839677 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:13.871743 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:13.890817 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:13.917423 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:13.919080 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:13.957633 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:14.014154 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:14.096959 15486 consensus_queue.cc:237] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:14.099606 15755 raft_consensus.cc:1261] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:14.100149 12861 raft_consensus.cc:1261] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:14.100709 23358 consensus_queue.cc:1035] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:14.100992 22986 consensus_queue.cc:1035] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:14.103868 23757 raft_consensus.cc:2932] T 5f705e37c6be455eaa2a19145474bf9e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:14.104755 15755 raft_consensus.cc:2932] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:14.116997 12586 catalog_manager.cc:5484] T 5f705e37c6be455eaa2a19145474bf9e P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:14.123867 12861 raft_consensus.cc:2932] T 5f705e37c6be455eaa2a19145474bf9e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:14.125171 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f705e37c6be455eaa2a19145474bf9e with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:14.126669 23767 tablet_replica.cc:330] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:14.126935 23767 raft_consensus.cc:2229] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:14.128846 23767 raft_consensus.cc:2258] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:14.129431 23767 ts_tablet_manager.cc:1875] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:14.135344 23767 ts_tablet_manager.cc:1888] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:14.135459 23767 log.cc:1196] T 5f705e37c6be455eaa2a19145474bf9e P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/5f705e37c6be455eaa2a19145474bf9e
I20240502 08:19:14.136200 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 5f705e37c6be455eaa2a19145474bf9e (table rebalancer_extra_table_20 [id=6a2ad006f2a4419ebb5b776a03711658]) successfully deleted
I20240502 08:19:14.371964 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:14.379889 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:14.397079 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:14.527541 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:14.527667 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:14.558240 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 08:19:14.594802 13056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 67315.9us
I20240502 08:19:14.611380 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:14.626675 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:14.786455 12860 consensus_queue.cc:237] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:14.788337 15486 raft_consensus.cc:1261] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:14.788467 15753 raft_consensus.cc:1261] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:14.788939 23627 consensus_queue.cc:1035] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:14.789196 23628 consensus_queue.cc:1035] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:14.791828 23627 raft_consensus.cc:2932] T 6546b77d0fd3434487dfe3abd335803e P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:14.792537 15486 raft_consensus.cc:2932] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:14.795205 15753 raft_consensus.cc:2932] T 6546b77d0fd3434487dfe3abd335803e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:14.800550 12585 catalog_manager.cc:5484] T 6546b77d0fd3434487dfe3abd335803e P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:14.818769 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 6546b77d0fd3434487dfe3abd335803e with delete_type TABLET_DATA_TOMBSTONED (TS 60b6e265e7754b7187307d05ef405dd4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38740
I20240502 08:19:14.819312 23851 tablet_replica.cc:330] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:19:14.819571 23851 raft_consensus.cc:2229] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:14.819857 23851 raft_consensus.cc:2258] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:14.820468 23851 ts_tablet_manager.cc:1875] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:14.828568 23851 ts_tablet_manager.cc:1888] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:14.828749 23851 log.cc:1196] T 6546b77d0fd3434487dfe3abd335803e P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/6546b77d0fd3434487dfe3abd335803e
I20240502 08:19:14.829731 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 6546b77d0fd3434487dfe3abd335803e (table rebalancer_extra_table_50 [id=ac4fff256cff4b5fbc1f66de9625967c]) successfully deleted
I20240502 08:19:15.038688 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:15.081895 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:15.085024 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:15.108377 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:15.121140 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:15.134914 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:15.144207 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:15.152855 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:15.300803 12861 consensus_queue.cc:237] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:15.302558 12991 raft_consensus.cc:1261] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:15.302565 15886 raft_consensus.cc:1261] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:15.303025 23628 consensus_queue.cc:1035] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:15.303262 23627 consensus_queue.cc:1035] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:15.308434 23628 raft_consensus.cc:2932] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:15.310072 15886 raft_consensus.cc:2932] T 581d2ed76e2e446eb84b94231ebb97f4 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:15.312060 12991 raft_consensus.cc:2932] T 581d2ed76e2e446eb84b94231ebb97f4 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:15.315083 12581 catalog_manager.cc:5484] T 581d2ed76e2e446eb84b94231ebb97f4 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:15.322336 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 581d2ed76e2e446eb84b94231ebb97f4 with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:19:15.322897 23933 tablet_replica.cc:330] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:19:15.323129 23933 raft_consensus.cc:2229] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:15.323369 23933 raft_consensus.cc:2258] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:15.325865 23933 ts_tablet_manager.cc:1875] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:15.339726 23933 ts_tablet_manager.cc:1888] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:15.339815 23933 log.cc:1196] T 581d2ed76e2e446eb84b94231ebb97f4 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/581d2ed76e2e446eb84b94231ebb97f4
I20240502 08:19:15.340430 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 581d2ed76e2e446eb84b94231ebb97f4 (table rebalance_test_table_19 [id=c7115c9eea2a450b944a15086ffe02bd]) successfully deleted
I20240502 08:19:15.575523 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:15.591267 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:15.606055 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:15.651955 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:15.684731 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:15.694844 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:15.702505 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:15.718081 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:15.887614 13122 consensus_queue.cc:237] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:15.889480 12861 raft_consensus.cc:1261] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:15.889561 12992 raft_consensus.cc:1261] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:15.890089 23986 consensus_queue.cc:1035] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:15.890347 23572 consensus_queue.cc:1035] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:15.892913 23986 raft_consensus.cc:2932] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:15.893759 12861 raft_consensus.cc:2932] T 087809a059d14b1f9a24ec3e956e1305 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:15.900064 12992 raft_consensus.cc:2932] T 087809a059d14b1f9a24ec3e956e1305 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:15.905539 12579 catalog_manager.cc:5484] T 087809a059d14b1f9a24ec3e956e1305 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:15.913609 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 087809a059d14b1f9a24ec3e956e1305 with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:15.930024 24020 tablet_replica.cc:330] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:15.930204 24020 raft_consensus.cc:2229] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:15.930373 24020 raft_consensus.cc:2258] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:15.930948 24020 ts_tablet_manager.cc:1875] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:15.935559 24020 ts_tablet_manager.cc:1888] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:15.935657 24020 log.cc:1196] T 087809a059d14b1f9a24ec3e956e1305 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/087809a059d14b1f9a24ec3e956e1305
I20240502 08:19:15.936407 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 087809a059d14b1f9a24ec3e956e1305 (table rebalancer_extra_table_48 [id=b1448b5e49614c108f8d76ba41e5da72]) successfully deleted
I20240502 08:19:16.173249 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:16.202703 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:16.219081 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:16.227614 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:16.299365 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:16.321161 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:16.416221 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:16.416441 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:16.547928 12991 consensus_queue.cc:237] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:16.549988 15753 raft_consensus.cc:1261] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:16.550189 13121 raft_consensus.cc:1261] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:16.550563 23424 consensus_queue.cc:1035] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:16.550849 23987 consensus_queue.cc:1035] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:16.553560 24096 raft_consensus.cc:2932] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:16.553972 13121 raft_consensus.cc:2932] T 5be106986ad247c597666d46a1dede7d P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:16.554075 15753 raft_consensus.cc:2932] T 5be106986ad247c597666d46a1dede7d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:16.559557 12587 catalog_manager.cc:5484] T 5be106986ad247c597666d46a1dede7d P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:16.567670 12708 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be106986ad247c597666d46a1dede7d with delete_type TABLET_DATA_TOMBSTONED (TS 29b035434087435cbaf4ebcac98eedee not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40606
I20240502 08:19:16.568218 24105 tablet_replica.cc:330] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:19:16.568735 24105 raft_consensus.cc:2229] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:16.569046 24105 raft_consensus.cc:2258] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:16.569672 24105 ts_tablet_manager.cc:1875] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:16.575872 24105 ts_tablet_manager.cc:1888] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:16.576012 24105 log.cc:1196] T 5be106986ad247c597666d46a1dede7d P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/5be106986ad247c597666d46a1dede7d
I20240502 08:19:16.576742 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 5be106986ad247c597666d46a1dede7d (table rebalancer_extra_table_58 [id=107c7945253947cda94a688a2c9373cd]) successfully deleted
I20240502 08:19:16.821133 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:16.827705 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:16.827904 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:16.875069 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:16.895843 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:16.917191 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:16.925904 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:16.926375 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:17.079586 12730 consensus_queue.cc:237] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:17.081610 15487 raft_consensus.cc:1261] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:17.081816 12861 raft_consensus.cc:1261] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:17.082362 24057 consensus_queue.cc:1035] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:17.082724 23677 consensus_queue.cc:1035] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:17.085220 24180 raft_consensus.cc:2932] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:17.085626 12861 raft_consensus.cc:2932] T 1f07b2caf1754c469a8e33382f615019 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:17.085691 15487 raft_consensus.cc:2932] T 1f07b2caf1754c469a8e33382f615019 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:17.091604 12582 catalog_manager.cc:5484] T 1f07b2caf1754c469a8e33382f615019 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:17.102684 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f07b2caf1754c469a8e33382f615019 with delete_type TABLET_DATA_TOMBSTONED (TS 60b6e265e7754b7187307d05ef405dd4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38740
I20240502 08:19:17.113971 24189 tablet_replica.cc:330] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:19:17.114349 24189 raft_consensus.cc:2229] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:17.114696 24189 raft_consensus.cc:2258] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:17.117264 24189 ts_tablet_manager.cc:1875] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:17.123698 24189 ts_tablet_manager.cc:1888] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:17.123823 24189 log.cc:1196] T 1f07b2caf1754c469a8e33382f615019 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/1f07b2caf1754c469a8e33382f615019
I20240502 08:19:17.124666 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 1f07b2caf1754c469a8e33382f615019 (table rebalance_test_table_1 [id=eafcd56a22d944b391206b320ed18540]) successfully deleted
W20240502 08:19:17.384703 13056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.5:44131, user_credentials={real_user=slave}} blocked reactor thread for 44945.7us
W20240502 08:19:17.401408 12926 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.2:36933, user_credentials={real_user=slave}} blocked reactor thread for 58477.7us
I20240502 08:19:17.567112 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:17.580941 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:17.632225 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:17.699896 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:17.833323 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:17.844074 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:17.861788 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:17.879747 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:18.006409 12991 consensus_queue.cc:237] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:18.008227 12861 raft_consensus.cc:1261] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:18.008304 15886 raft_consensus.cc:1261] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:18.008705 24232 consensus_queue.cc:1035] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:18.009191 24231 consensus_queue.cc:1035] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:18.011545 24232 raft_consensus.cc:2932] T 660b389a179f49eeb3007dea6ce078e9 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:18.015125 15886 raft_consensus.cc:2932] T 660b389a179f49eeb3007dea6ce078e9 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:18.020409 12861 raft_consensus.cc:2932] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:18.027406 12583 catalog_manager.cc:5484] T 660b389a179f49eeb3007dea6ce078e9 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:18.064004 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 660b389a179f49eeb3007dea6ce078e9 with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:19:18.080816 24290 tablet_replica.cc:330] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:19:18.081032 24290 raft_consensus.cc:2229] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:18.081279 24290 raft_consensus.cc:2258] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:18.082141 24290 ts_tablet_manager.cc:1875] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:18.089423 24290 ts_tablet_manager.cc:1888] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:18.089542 24290 log.cc:1196] T 660b389a179f49eeb3007dea6ce078e9 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/660b389a179f49eeb3007dea6ce078e9
I20240502 08:19:18.090665 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 660b389a179f49eeb3007dea6ce078e9 (table rebalance_test_table_15 [id=f13587d289664cc99887ffe5c7242379]) successfully deleted
I20240502 08:19:18.295934 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:18.298365 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:18.310632 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:18.324630 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:18.331434 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:18.333092 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:18.361933 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:18.398770 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:18.557941 12860 consensus_queue.cc:237] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:18.560040 15755 raft_consensus.cc:1261] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:18.560118 13121 raft_consensus.cc:1261] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:18.560526 24213 consensus_queue.cc:1035] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:18.560742 24011 consensus_queue.cc:1035] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:18.563272 24213 raft_consensus.cc:2932] T c631a6dbc0d44f76abf1cf5a14ea4a6d P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:18.563908 15755 raft_consensus.cc:2932] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:18.569862 12579 catalog_manager.cc:5484] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:18.578248 13121 raft_consensus.cc:2932] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:18.584425 12708 tablet_service.cc:1514] Processing DeleteTablet for tablet c631a6dbc0d44f76abf1cf5a14ea4a6d with delete_type TABLET_DATA_TOMBSTONED (TS 29b035434087435cbaf4ebcac98eedee not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40606
I20240502 08:19:18.585047 24365 tablet_replica.cc:330] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:19:18.585291 24365 raft_consensus.cc:2229] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:18.585830 24365 raft_consensus.cc:2258] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:18.586547 24365 ts_tablet_manager.cc:1875] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:18.591876 24365 ts_tablet_manager.cc:1888] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:18.591993 24365 log.cc:1196] T c631a6dbc0d44f76abf1cf5a14ea4a6d P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/c631a6dbc0d44f76abf1cf5a14ea4a6d
I20240502 08:19:18.592701 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet c631a6dbc0d44f76abf1cf5a14ea4a6d (table rebalancer_extra_table_46 [id=7b4c833a5fb54b3e967ac5b77b0558d3]) successfully deleted
I20240502 08:19:18.866598 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:18.910167 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:18.924239 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:18.953436 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:18.967445 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:19.010403 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:19.010883 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:19.055660 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:19.167435 15753 consensus_queue.cc:237] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:19.169271 15886 raft_consensus.cc:1261] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:19:19.169409 12861 raft_consensus.cc:1261] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:19:19.169796 24247 consensus_queue.cc:1035] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:19.170025 24252 consensus_queue.cc:1035] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:19.172668 24247 raft_consensus.cc:2932] T 040878c4c4ae4684b02296ee9eac8985 P 9333cb10227b4801b4fa7c24e22e131b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:19.173175 15886 raft_consensus.cc:2932] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:19.178527 12579 catalog_manager.cc:5484] T 040878c4c4ae4684b02296ee9eac8985 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 2 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:19.181643 12861 raft_consensus.cc:2932] T 040878c4c4ae4684b02296ee9eac8985 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:19.194032 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 040878c4c4ae4684b02296ee9eac8985 with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:19.194594 24447 tablet_replica.cc:330] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:19.194828 24447 raft_consensus.cc:2229] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:19.195032 24447 raft_consensus.cc:2258] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:19.195577 24447 ts_tablet_manager.cc:1875] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:19.202283 24447 ts_tablet_manager.cc:1888] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.2
I20240502 08:19:19.202571 24447 log.cc:1196] T 040878c4c4ae4684b02296ee9eac8985 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/040878c4c4ae4684b02296ee9eac8985
I20240502 08:19:19.203274 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 040878c4c4ae4684b02296ee9eac8985 (table rebalancer_extra_table_2 [id=ab84e3c981dc4fcf95f4296f05da39ba]) successfully deleted
I20240502 08:19:19.463161 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:19.463264 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:19.471201 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:19.508069 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:19.540344 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:19.540728 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:19.644264 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:19.683066 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:19.802742 12730 consensus_queue.cc:237] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:19.804404 15886 raft_consensus.cc:1261] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:19.804490 12861 raft_consensus.cc:1261] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:19.804960 24180 consensus_queue.cc:1035] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:19.805202 24394 consensus_queue.cc:1035] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:19.807844 24180 raft_consensus.cc:2932] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:19.811736 12861 raft_consensus.cc:2932] T 1fba813ac35c4382a0917604c2057496 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:19.815780 15886 raft_consensus.cc:2932] T 1fba813ac35c4382a0917604c2057496 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:19.821760 12582 catalog_manager.cc:5484] T 1fba813ac35c4382a0917604c2057496 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:19.827832 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fba813ac35c4382a0917604c2057496 with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:19:19.828421 24531 tablet_replica.cc:330] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:19:19.828660 24531 raft_consensus.cc:2229] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:19.834801 24531 raft_consensus.cc:2258] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:19.835348 24531 ts_tablet_manager.cc:1875] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:19.840361 24531 ts_tablet_manager.cc:1888] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:19.840922 24531 log.cc:1196] T 1fba813ac35c4382a0917604c2057496 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/1fba813ac35c4382a0917604c2057496
I20240502 08:19:19.841687 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 1fba813ac35c4382a0917604c2057496 (table rebalance_test_table_17 [id=c8304b52e21948eba4b4290948c74b5d]) successfully deleted
I20240502 08:19:19.889906 12728 consensus_queue.cc:237] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:19.892144 13120 raft_consensus.cc:1261] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:19.892751 12991 raft_consensus.cc:1261] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:19.892923 24394 consensus_queue.cc:1035] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:19.893316 24556 consensus_queue.cc:1035] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:19:19.893762 12664 consensus_peers.cc:485] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee -> Peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Couldn't send request to peer db7893bb29a047b3b82cc60bfd8930a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b40f3044cd6485ca72b05873dc93f7e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:19.896181 24394 raft_consensus.cc:2932] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:19.896787 12992 raft_consensus.cc:2932] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:19.901491 13120 raft_consensus.cc:2932] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:19.902379 12586 catalog_manager.cc:5484] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:19.938618 12727 consensus_queue.cc:237] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:19.940701 12861 raft_consensus.cc:1261] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:19.940865 15886 raft_consensus.cc:1261] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:19.941002 12663 consensus_peers.cc:485] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee -> Peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Couldn't send request to peer 9333cb10227b4801b4fa7c24e22e131b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c24538742b874c92a70e17b3450993fb. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:19.941300 24180 consensus_queue.cc:1035] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:19.942113 24180 consensus_queue.cc:1035] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:19.944695 24394 raft_consensus.cc:2932] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:19.945186 15886 raft_consensus.cc:2932] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:19.949642 12861 raft_consensus.cc:2932] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:19.953024 12586 catalog_manager.cc:5484] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index -1 to 2, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:19.989420 15622 consensus_queue.cc:237] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
W20240502 08:19:19.991803 15554 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b -> Peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): Couldn't send request to peer 60b6e265e7754b7187307d05ef405dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fcbd19e0a9b49eaa3e403683fe6b34b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:19.991847 15886 raft_consensus.cc:1261] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:19.992302 24147 consensus_queue.cc:1035] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:19.993846 15484 raft_consensus.cc:1261] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:19.994349 24161 consensus_queue.cc:1035] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:19.999547 24147 raft_consensus.cc:2932] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:20.000094 15484 raft_consensus.cc:2932] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:20.004284 15886 raft_consensus.cc:2932] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:20.006608 12583 catalog_manager.cc:5484] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:20.309844 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:20.357136 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:20.357674 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:20.364099 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:20.367439 24658 ts_tablet_manager.cc:902] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:20.386595 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:20.406247 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:20.436483 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:20.454113 24661 ts_tablet_manager.cc:902] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4: Initiating tablet copy from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:20.460089 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:20.460587 24662 ts_tablet_manager.cc:902] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:20.548512 24661 tablet_copy_client.cc:308] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.6:44983
I20240502 08:19:20.549224 15642 tablet_copy_service.cc:140] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Received BeginTabletCopySession request for tablet 5fcbd19e0a9b49eaa3e403683fe6b34b from peer 60b6e265e7754b7187307d05ef405dd4 ({username='slave'} at 127.6.5.3:49097)
I20240502 08:19:20.549350 15642 tablet_copy_service.cc:161] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Beginning new tablet copy session on tablet 5fcbd19e0a9b49eaa3e403683fe6b34b from peer 60b6e265e7754b7187307d05ef405dd4 at {username='slave'} at 127.6.5.3:49097: session id = 60b6e265e7754b7187307d05ef405dd4-5fcbd19e0a9b49eaa3e403683fe6b34b
I20240502 08:19:20.552060 15642 tablet_copy_source_session.cc:205] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:20.552836 24662 tablet_copy_client.cc:308] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:19:20.557046 24658 tablet_copy_client.cc:308] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:19:20.557672 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet 6b40f3044cd6485ca72b05873dc93f7e from peer db7893bb29a047b3b82cc60bfd8930a3 ({username='slave'} at 127.6.5.5:46815)
I20240502 08:19:20.557839 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet 6b40f3044cd6485ca72b05873dc93f7e from peer db7893bb29a047b3b82cc60bfd8930a3 at {username='slave'} at 127.6.5.5:46815: session id = db7893bb29a047b3b82cc60bfd8930a3-6b40f3044cd6485ca72b05873dc93f7e
I20240502 08:19:20.559826 24661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcbd19e0a9b49eaa3e403683fe6b34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:20.560343 12751 tablet_copy_source_session.cc:205] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:20.561753 24658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b40f3044cd6485ca72b05873dc93f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:20.569986 24661 tablet_copy_client.cc:783] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:20.570278 24661 tablet_copy_client.cc:647] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:20.572417 24658 tablet_copy_client.cc:783] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:20.572700 24658 tablet_copy_client.cc:647] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:20.572798 24661 tablet_copy_client.cc:523] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:20.574734 24658 tablet_copy_client.cc:523] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:20.577350 24661 tablet_bootstrap.cc:492] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:19:20.578158 24658 tablet_bootstrap.cc:492] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
W20240502 08:19:20.582525 15688 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.2:36933, user_credentials={real_user=slave}} blocked reactor thread for 58079us
I20240502 08:19:20.583144 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet c24538742b874c92a70e17b3450993fb from peer 9333cb10227b4801b4fa7c24e22e131b ({username='slave'} at 127.6.5.7:35555)
I20240502 08:19:20.583318 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet c24538742b874c92a70e17b3450993fb from peer 9333cb10227b4801b4fa7c24e22e131b at {username='slave'} at 127.6.5.7:35555: session id = 9333cb10227b4801b4fa7c24e22e131b-c24538742b874c92a70e17b3450993fb
I20240502 08:19:20.586104 12751 tablet_copy_source_session.cc:205] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 08:19:20.586373 15690 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.6:44983, user_credentials={real_user=slave}} blocked reactor thread for 66473us
I20240502 08:19:20.601773 24662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24538742b874c92a70e17b3450993fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:20.608273 24661 tablet_bootstrap.cc:492] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:20.608511 24662 tablet_copy_client.cc:783] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:20.608680 24661 tablet_bootstrap.cc:492] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4: Bootstrap complete.
I20240502 08:19:20.608721 24662 tablet_copy_client.cc:647] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:20.608979 24661 ts_tablet_manager.cc:1367] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.009s
I20240502 08:19:20.609699 24661 raft_consensus.cc:348] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:20.609959 24661 raft_consensus.cc:729] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:19:20.610316 24662 tablet_copy_client.cc:523] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:20.610209 24661 consensus_queue.cc:260] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:20.610783 24661 ts_tablet_manager.cc:1398] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:19:20.611954 15642 tablet_copy_service.cc:342] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Request end of tablet copy session 60b6e265e7754b7187307d05ef405dd4-5fcbd19e0a9b49eaa3e403683fe6b34b received from {username='slave'} at 127.6.5.3:49097
I20240502 08:19:20.612056 15642 tablet_copy_service.cc:433] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: ending tablet copy session 60b6e265e7754b7187307d05ef405dd4-5fcbd19e0a9b49eaa3e403683fe6b34b on tablet 5fcbd19e0a9b49eaa3e403683fe6b34b with peer 60b6e265e7754b7187307d05ef405dd4
I20240502 08:19:20.613785 24662 tablet_bootstrap.cc:492] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:19:20.645445 24662 tablet_bootstrap.cc:492] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:20.645967 24662 tablet_bootstrap.cc:492] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap complete.
I20240502 08:19:20.646306 24662 ts_tablet_manager.cc:1367] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.018s
I20240502 08:19:20.647284 24662 raft_consensus.cc:348] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:20.647619 24662 raft_consensus.cc:729] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:19:20.648268 24658 tablet_bootstrap.cc:492] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:20.648442 24662 consensus_queue.cc:260] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:20.648874 24658 tablet_bootstrap.cc:492] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap complete.
I20240502 08:19:20.649158 24658 ts_tablet_manager.cc:1367] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.071s user 0.046s sys 0.004s
I20240502 08:19:20.650509 24658 raft_consensus.cc:348] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:20.650813 24658 raft_consensus.cc:729] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:19:20.651085 24658 consensus_queue.cc:260] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:20.651962 24658 ts_tablet_manager.cc:1398] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 08:19:20.652738 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session db7893bb29a047b3b82cc60bfd8930a3-6b40f3044cd6485ca72b05873dc93f7e received from {username='slave'} at 127.6.5.5:46815
I20240502 08:19:20.652868 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session db7893bb29a047b3b82cc60bfd8930a3-6b40f3044cd6485ca72b05873dc93f7e on tablet 6b40f3044cd6485ca72b05873dc93f7e with peer db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:19:20.653534 24662 ts_tablet_manager.cc:1398] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:19:20.659477 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 9333cb10227b4801b4fa7c24e22e131b-c24538742b874c92a70e17b3450993fb received from {username='slave'} at 127.6.5.7:35555
I20240502 08:19:20.659597 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 9333cb10227b4801b4fa7c24e22e131b-c24538742b874c92a70e17b3450993fb on tablet c24538742b874c92a70e17b3450993fb with peer 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:19:20.734779 15484 raft_consensus.cc:1203] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:20.979579 15755 raft_consensus.cc:1203] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:20.993880 12991 raft_consensus.cc:1203] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:20.997659 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:21.002717 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:21.019316 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:21.019337 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:21.041316 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:21.055073 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:21.055737 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:19:21.081753 12664 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.2:36933, user_credentials={real_user=slave}} blocked reactor thread for 46249.4us
I20240502 08:19:21.083091 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240502 08:19:21.100731 12795 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.7:38851, user_credentials={real_user=slave}} blocked reactor thread for 82986.8us
I20240502 08:19:21.231290 15621 consensus_queue.cc:237] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:21.233397 15885 raft_consensus.cc:1261] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:21.233417 12992 raft_consensus.cc:1261] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:21.234009 24147 consensus_queue.cc:1035] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:21.235328 24147 consensus_queue.cc:1035] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:21.237735 24161 raft_consensus.cc:2932] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:21.238430 15885 raft_consensus.cc:2932] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:21.246007 12588 catalog_manager.cc:5484] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:21.252218 12992 raft_consensus.cc:2932] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:21.261003 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcbd19e0a9b49eaa3e403683fe6b34b with delete_type TABLET_DATA_TOMBSTONED (TS db7893bb29a047b3b82cc60bfd8930a3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46880
I20240502 08:19:21.261536 24770 tablet_replica.cc:330] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:19:21.261837 24770 raft_consensus.cc:2229] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:21.262163 24770 raft_consensus.cc:2258] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:21.262887 24770 ts_tablet_manager.cc:1875] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:21.267699 24770 ts_tablet_manager.cc:1888] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:21.267802 24770 log.cc:1196] T 5fcbd19e0a9b49eaa3e403683fe6b34b P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/5fcbd19e0a9b49eaa3e403683fe6b34b
I20240502 08:19:21.268409 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 5fcbd19e0a9b49eaa3e403683fe6b34b (table rebalancer_extra_table_20 [id=6a2ad006f2a4419ebb5b776a03711658]) successfully deleted
I20240502 08:19:21.495617 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:21.515101 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:21.538597 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:21.544272 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:21.559813 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:21.586427 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:21.595448 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:21.657933 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:21.805291 12727 consensus_queue.cc:237] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:21.807113 13120 raft_consensus.cc:1261] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.807382 15488 raft_consensus.cc:1261] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.807597 24745 consensus_queue.cc:1035] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.807817 24829 consensus_queue.cc:1035] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.810422 24829 raft_consensus.cc:2932] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:21.813558 13120 raft_consensus.cc:2932] T 6b40f3044cd6485ca72b05873dc93f7e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:21.818307 12586 catalog_manager.cc:5484] T 6b40f3044cd6485ca72b05873dc93f7e P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:21.826910 15488 raft_consensus.cc:2932] T 6b40f3044cd6485ca72b05873dc93f7e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:21.838997 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b40f3044cd6485ca72b05873dc93f7e with delete_type TABLET_DATA_TOMBSTONED (TS 60b6e265e7754b7187307d05ef405dd4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38740
I20240502 08:19:21.840214 24857 tablet_replica.cc:330] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:19:21.840461 24857 raft_consensus.cc:2229] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:21.840746 24857 raft_consensus.cc:2258] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:21.841461 24857 ts_tablet_manager.cc:1875] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:21.846778 24857 ts_tablet_manager.cc:1888] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:21.846956 24857 log.cc:1196] T 6b40f3044cd6485ca72b05873dc93f7e P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/6b40f3044cd6485ca72b05873dc93f7e
I20240502 08:19:21.847668 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 6b40f3044cd6485ca72b05873dc93f7e (table rebalance_test_table_15 [id=f13587d289664cc99887ffe5c7242379]) successfully deleted
I20240502 08:19:22.075212 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:22.094616 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:22.106045 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:22.123829 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:22.210409 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:22.227794 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:22.239176 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:22.263689 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:22.411584 15885 consensus_queue.cc:237] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:22.414050 12728 raft_consensus.cc:1261] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:22.414187 13122 raft_consensus.cc:1261] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:22.414630 24501 consensus_queue.cc:1035] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:22.415220 24336 consensus_queue.cc:1035] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:22.417116 24501 raft_consensus.cc:2932] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:22.418416 12728 raft_consensus.cc:2932] T 0d57fc923fe946afa65a6aa8ce52ba7b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:22.418584 13120 raft_consensus.cc:2932] T 0d57fc923fe946afa65a6aa8ce52ba7b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:22.423331 12581 catalog_manager.cc:5484] T 0d57fc923fe946afa65a6aa8ce52ba7b P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index 2 to 3, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:22.444255 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d57fc923fe946afa65a6aa8ce52ba7b with delete_type TABLET_DATA_TOMBSTONED (TS e1dd14f7ca764a1490df419f13f2804a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:56258
I20240502 08:19:22.463881 24939 tablet_replica.cc:330] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:19:22.464089 24939 raft_consensus.cc:2229] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:22.464277 24939 raft_consensus.cc:2258] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:22.464883 24939 ts_tablet_manager.cc:1875] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:22.469285 24939 ts_tablet_manager.cc:1888] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:22.478431 24939 log.cc:1196] T 0d57fc923fe946afa65a6aa8ce52ba7b P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/0d57fc923fe946afa65a6aa8ce52ba7b
I20240502 08:19:22.479506 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 0d57fc923fe946afa65a6aa8ce52ba7b (table rebalancer_extra_table_14 [id=70609143f5404be1badc9b37bce65793]) successfully deleted
I20240502 08:19:22.651229 12992 consensus_queue.cc:237] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:22.660508 13120 raft_consensus.cc:1261] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:22.662422 24276 consensus_queue.cc:1035] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:19:22.663775 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 553cec4eb28341dba84afc79b540ae4b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:22.664434 12728 raft_consensus.cc:1261] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:22.665563 24609 consensus_queue.cc:1035] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:22.675467 24609 raft_consensus.cc:2932] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:22.676026 13120 raft_consensus.cc:2932] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:22.680241 12728 raft_consensus.cc:2932] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:22.681121 12582 catalog_manager.cc:5484] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:22.728865 15622 consensus_queue.cc:237] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:22.731360 15755 raft_consensus.cc:1261] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:22.731400 15886 raft_consensus.cc:1261] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:22.732074 25046 consensus_queue.cc:1035] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:19:22.732357 15554 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8de012363e4f4455acd96cb674ab0c4b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:22.732630 24764 consensus_queue.cc:1035] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:22.735877 24764 raft_consensus.cc:2932] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:22.736410 15755 raft_consensus.cc:2932] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:22.739545 15886 raft_consensus.cc:2932] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:22.744710 12581 catalog_manager.cc:5484] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:22.819805 12991 consensus_queue.cc:237] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
W20240502 08:19:22.822074 12925 consensus_peers.cc:485] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 -> Peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Couldn't send request to peer 6a0b05f3250d415590b0edf6b44fbe1f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d3592822ca849b9a47c4320b84754ae. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:22.822250 15886 raft_consensus.cc:1261] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:22.822389 15488 raft_consensus.cc:1261] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:22.822813 25026 consensus_queue.cc:1035] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:22.823438 25026 consensus_queue.cc:1035] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:22.826247 24276 raft_consensus.cc:2932] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:22.826874 15488 raft_consensus.cc:2932] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:22.831389 15886 raft_consensus.cc:2932] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:22.833544 12581 catalog_manager.cc:5484] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:22.880129 12860 consensus_queue.cc:237] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:22.882387 15621 raft_consensus.cc:1261] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:22.882982 12796 consensus_peers.cc:485] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a -> Peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Couldn't send request to peer 29b035434087435cbaf4ebcac98eedee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2ad585a32864067a5d8eaea5f8260f1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:22.883702 15755 raft_consensus.cc:1261] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:22.883771 24355 consensus_queue.cc:1035] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:22.885048 25085 consensus_queue.cc:1035] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:22.887467 24355 raft_consensus.cc:2932] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:22.887943 15755 raft_consensus.cc:2932] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:22.890969 15621 raft_consensus.cc:2932] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:22.893530 12581 catalog_manager.cc:5484] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:22.935338 12727 consensus_queue.cc:237] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:22.937950 15886 raft_consensus.cc:1261] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:22.938588 15488 raft_consensus.cc:1261] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:22.939097 24829 consensus_queue.cc:1035] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:22.940611 24829 consensus_queue.cc:1035] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:19:22.942992 12664 consensus_peers.cc:485] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee -> Peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): Couldn't send request to peer 60b6e265e7754b7187307d05ef405dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ae7246870f14d7da744a1856430487c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:22.955430 24829 raft_consensus.cc:2932] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:22.959226 15886 raft_consensus.cc:2932] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:22.959156 15488 raft_consensus.cc:2932] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:22.961347 12584 catalog_manager.cc:5484] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:22.994520 13122 consensus_queue.cc:237] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:22.996496 15488 raft_consensus.cc:1261] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:22.996721 15755 raft_consensus.cc:1261] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:22.996778 13056 consensus_peers.cc:485] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e141ca70856946a1a3f8e21b77548736. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:22.997032 24911 consensus_queue.cc:1035] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:22.997591 24911 consensus_queue.cc:1035] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:23.000062 24665 raft_consensus.cc:2932] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:23.000723 15755 raft_consensus.cc:2932] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:23.004351 15488 raft_consensus.cc:2932] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:23.007416 12581 catalog_manager.cc:5484] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:23.059571 15488 consensus_queue.cc:237] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:23.061964 12727 raft_consensus.cc:1261] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:23.062212 15885 raft_consensus.cc:1261] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:23.062450 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9ad7bf820c84aae9a8875fb92f8e5d5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:23.063004 25059 consensus_queue.cc:1035] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:23.063275 24608 consensus_queue.cc:1035] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:23.073931 25059 raft_consensus.cc:2932] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:23.074913 12727 raft_consensus.cc:2932] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:23.080767 12584 catalog_manager.cc:5484] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:23.090015 15885 raft_consensus.cc:2932] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:23.125489 12727 consensus_queue.cc:237] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:23.127642 13122 raft_consensus.cc:1261] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:23.128194 12991 raft_consensus.cc:1261] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:23.128172 24829 consensus_queue.cc:1035] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:23.128700 24829 consensus_queue.cc:1035] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:19:23.128949 12664 consensus_peers.cc:485] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee -> Peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Couldn't send request to peer db7893bb29a047b3b82cc60bfd8930a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9366b55ca7dc48bfa5d2ed650d7a30e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:23.131006 25105 raft_consensus.cc:2932] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:23.131502 12991 raft_consensus.cc:2932] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:23.134690 13122 raft_consensus.cc:2932] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:23.140755 12582 catalog_manager.cc:5484] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:23.171281 15622 consensus_queue.cc:237] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:23.173296 15885 raft_consensus.cc:1261] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:23.173458 15554 consensus_peers.cc:485] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b -> Peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Couldn't send request to peer 29b035434087435cbaf4ebcac98eedee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10ba9ba78b5444a4bc73be4f6cfd6370. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:23.173893 25046 consensus_queue.cc:1035] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:23.174160 15753 raft_consensus.cc:1261] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:23.175932 25046 consensus_queue.cc:1035] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:23.178498 24850 raft_consensus.cc:2932] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:23.179044 15753 raft_consensus.cc:2932] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:23.182412 15885 raft_consensus.cc:2932] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:23.188917 12582 catalog_manager.cc:5484] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:23.205222 25186 ts_tablet_manager.cc:902] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:23.226802 25186 tablet_copy_client.cc:308] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:19:23.227366 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet 553cec4eb28341dba84afc79b540ae4b from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:58785)
I20240502 08:19:23.227488 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet 553cec4eb28341dba84afc79b540ae4b from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:58785: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-553cec4eb28341dba84afc79b540ae4b
I20240502 08:19:23.229249 13012 tablet_copy_source_session.cc:205] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.231209 25186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553cec4eb28341dba84afc79b540ae4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.237776 25186 tablet_copy_client.cc:783] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.238489 25186 tablet_copy_client.cc:647] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.239915 25186 tablet_copy_client.cc:523] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.242918 25186 tablet_bootstrap.cc:492] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:19:23.257405 25201 ts_tablet_manager.cc:902] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:23.282845 25201 tablet_copy_client.cc:308] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.6:44983
I20240502 08:19:23.283403 15642 tablet_copy_service.cc:140] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Received BeginTabletCopySession request for tablet 8de012363e4f4455acd96cb674ab0c4b from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:42935)
I20240502 08:19:23.283561 15642 tablet_copy_service.cc:161] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Beginning new tablet copy session on tablet 8de012363e4f4455acd96cb674ab0c4b from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:42935: session id = e1dd14f7ca764a1490df419f13f2804a-8de012363e4f4455acd96cb674ab0c4b
I20240502 08:19:23.286293 15642 tablet_copy_source_session.cc:205] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.287653 25201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de012363e4f4455acd96cb674ab0c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.293833 25201 tablet_copy_client.cc:783] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.294109 25201 tablet_copy_client.cc:647] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.296381 25201 tablet_copy_client.cc:523] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.298739 25186 tablet_bootstrap.cc:492] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.299137 25186 tablet_bootstrap.cc:492] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:19:23.299896 25186 ts_tablet_manager.cc:1367] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.057s user 0.043s sys 0.001s
I20240502 08:19:23.300565 25186 raft_consensus.cc:348] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:23.300798 25186 raft_consensus.cc:729] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:19:23.301013 25186 consensus_queue.cc:260] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:23.301537 25186 ts_tablet_manager.cc:1398] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:19:23.302218 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-553cec4eb28341dba84afc79b540ae4b received from {username='slave'} at 127.6.5.6:58785
I20240502 08:19:23.302336 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-553cec4eb28341dba84afc79b540ae4b on tablet 553cec4eb28341dba84afc79b540ae4b with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:23.310150 25201 tablet_bootstrap.cc:492] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:19:23.317495 12860 consensus_queue.cc:237] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:23.319788 25201 tablet_bootstrap.cc:492] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.320195 25201 tablet_bootstrap.cc:492] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:19:23.320448 25201 ts_tablet_manager.cc:1367] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
W20240502 08:19:23.320570 12797 consensus_peers.cc:485] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a -> Peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Couldn't send request to peer 10f0e36203914b85b77e78609ffced98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21f25468f1b541dc9ede27359acc2b9c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:23.320585 12991 raft_consensus.cc:1261] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:23.320691 12727 raft_consensus.cc:1261] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:23.321215 25088 consensus_queue.cc:1035] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:23.321166 25201 raft_consensus.cc:348] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:23.321377 25201 raft_consensus.cc:729] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:19:23.321475 25085 consensus_queue.cc:1035] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:23.321596 25201 consensus_queue.cc:260] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:23.322113 25201 ts_tablet_manager.cc:1398] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:19:23.322772 15642 tablet_copy_service.cc:342] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-8de012363e4f4455acd96cb674ab0c4b received from {username='slave'} at 127.6.5.2:42935
I20240502 08:19:23.322901 15642 tablet_copy_service.cc:433] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-8de012363e4f4455acd96cb674ab0c4b on tablet 8de012363e4f4455acd96cb674ab0c4b with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:23.328172 25088 raft_consensus.cc:2932] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:23.328568 12727 raft_consensus.cc:2932] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:23.334216 12991 raft_consensus.cc:2932] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:23.334343 12582 catalog_manager.cc:5484] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index -1 to 4, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:23.399689 15488 consensus_queue.cc:237] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:23.401796 12727 raft_consensus.cc:1261] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:23.401975 15419 consensus_peers.cc:485] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 646aec80c3634b6ba8493832ff4294a0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:23.401970 15754 raft_consensus.cc:1261] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:23.402359 25146 consensus_queue.cc:1035] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:23.402731 25059 consensus_queue.cc:1035] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:23.407526 25146 raft_consensus.cc:2932] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:23.407938 15754 raft_consensus.cc:2932] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:23.412006 12728 raft_consensus.cc:2932] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:23.414439 12582 catalog_manager.cc:5484] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:23.427807 25237 ts_tablet_manager.cc:902] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:23.431815 25235 ts_tablet_manager.cc:902] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee: Initiating tablet copy from peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:19:23.432998 25237 tablet_copy_client.cc:308] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:19:23.433758 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet 1d3592822ca849b9a47c4320b84754ae from peer 6a0b05f3250d415590b0edf6b44fbe1f ({username='slave'} at 127.6.5.4:39793)
I20240502 08:19:23.433912 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet 1d3592822ca849b9a47c4320b84754ae from peer 6a0b05f3250d415590b0edf6b44fbe1f at {username='slave'} at 127.6.5.4:39793: session id = 6a0b05f3250d415590b0edf6b44fbe1f-1d3592822ca849b9a47c4320b84754ae
I20240502 08:19:23.436419 13012 tablet_copy_source_session.cc:205] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.437577 25235 tablet_copy_client.cc:308] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:36933
I20240502 08:19:23.437819 25237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3592822ca849b9a47c4320b84754ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.438091 12882 tablet_copy_service.cc:140] P e1dd14f7ca764a1490df419f13f2804a: Received BeginTabletCopySession request for tablet c2ad585a32864067a5d8eaea5f8260f1 from peer 29b035434087435cbaf4ebcac98eedee ({username='slave'} at 127.6.5.1:56455)
I20240502 08:19:23.438258 12882 tablet_copy_service.cc:161] P e1dd14f7ca764a1490df419f13f2804a: Beginning new tablet copy session on tablet c2ad585a32864067a5d8eaea5f8260f1 from peer 29b035434087435cbaf4ebcac98eedee at {username='slave'} at 127.6.5.1:56455: session id = 29b035434087435cbaf4ebcac98eedee-c2ad585a32864067a5d8eaea5f8260f1
I20240502 08:19:23.440349 12882 tablet_copy_source_session.cc:205] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.441984 25250 ts_tablet_manager.cc:902] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:23.445662 25237 tablet_copy_client.cc:783] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.445852 25237 tablet_copy_client.cc:647] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.447749 25237 tablet_copy_client.cc:523] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.453933 25201 ts_tablet_manager.cc:902] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:19:23.461869 25235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ad585a32864067a5d8eaea5f8260f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.477104 25235 tablet_copy_client.cc:783] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.477376 25235 tablet_copy_client.cc:647] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.479190 25235 tablet_copy_client.cc:523] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.483623 25237 tablet_bootstrap.cc:492] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:19:23.484563 25235 tablet_bootstrap.cc:492] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:19:23.495201 25201 tablet_copy_client.cc:308] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:19:23.495842 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet e141ca70856946a1a3f8e21b77548736 from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:36987)
I20240502 08:19:23.496006 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet e141ca70856946a1a3f8e21b77548736 from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:36987: session id = e1dd14f7ca764a1490df419f13f2804a-e141ca70856946a1a3f8e21b77548736
I20240502 08:19:23.498746 13142 tablet_copy_source_session.cc:205] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.499866 25201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e141ca70856946a1a3f8e21b77548736. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.501586 25235 tablet_bootstrap.cc:492] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.501967 25235 tablet_bootstrap.cc:492] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee: Bootstrap complete.
I20240502 08:19:23.502313 25235 ts_tablet_manager.cc:1367] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.005s
I20240502 08:19:23.503091 25235 raft_consensus.cc:348] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:23.505195 25201 tablet_copy_client.cc:783] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.505424 25201 tablet_copy_client.cc:647] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.503327 25235 raft_consensus.cc:729] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:19:23.506392 25250 tablet_copy_client.cc:308] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:19:23.506307 25235 consensus_queue.cc:260] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:23.507493 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet 4ae7246870f14d7da744a1856430487c from peer 60b6e265e7754b7187307d05ef405dd4 ({username='slave'} at 127.6.5.3:52517)
I20240502 08:19:23.507637 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet 4ae7246870f14d7da744a1856430487c from peer 60b6e265e7754b7187307d05ef405dd4 at {username='slave'} at 127.6.5.3:52517: session id = 60b6e265e7754b7187307d05ef405dd4-4ae7246870f14d7da744a1856430487c
I20240502 08:19:23.510337 12751 tablet_copy_source_session.cc:205] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.511493 25235 ts_tablet_manager.cc:1398] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240502 08:19:23.512916 25201 tablet_copy_client.cc:523] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.517124 25237 tablet_bootstrap.cc:492] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.517500 25237 tablet_bootstrap.cc:492] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap complete.
I20240502 08:19:23.517925 25237 ts_tablet_manager.cc:1367] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.004s
I20240502 08:19:23.518199 12882 tablet_copy_service.cc:342] P e1dd14f7ca764a1490df419f13f2804a: Request end of tablet copy session 29b035434087435cbaf4ebcac98eedee-c2ad585a32864067a5d8eaea5f8260f1 received from {username='slave'} at 127.6.5.1:56455
I20240502 08:19:23.518324 12882 tablet_copy_service.cc:433] P e1dd14f7ca764a1490df419f13f2804a: ending tablet copy session 29b035434087435cbaf4ebcac98eedee-c2ad585a32864067a5d8eaea5f8260f1 on tablet c2ad585a32864067a5d8eaea5f8260f1 with peer 29b035434087435cbaf4ebcac98eedee
I20240502 08:19:23.518993 25237 raft_consensus.cc:348] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:23.519208 25237 raft_consensus.cc:729] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:19:23.519424 25237 consensus_queue.cc:260] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:23.520144 25201 tablet_bootstrap.cc:492] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:19:23.521092 25237 ts_tablet_manager.cc:1398] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:19:23.521720 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-1d3592822ca849b9a47c4320b84754ae received from {username='slave'} at 127.6.5.4:39793
I20240502 08:19:23.521837 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-1d3592822ca849b9a47c4320b84754ae on tablet 1d3592822ca849b9a47c4320b84754ae with peer 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:19:23.524394 25250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae7246870f14d7da744a1856430487c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.531872 25250 tablet_copy_client.cc:783] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.532071 25250 tablet_copy_client.cc:647] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.533488 25250 tablet_copy_client.cc:523] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.536924 25250 tablet_bootstrap.cc:492] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:19:23.538815 25201 tablet_bootstrap.cc:492] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.539242 25201 tablet_bootstrap.cc:492] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:19:23.539582 25201 ts_tablet_manager.cc:1367] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.005s
I20240502 08:19:23.540453 25201 raft_consensus.cc:348] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:23.542258 25201 raft_consensus.cc:729] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:19:23.542665 25201 consensus_queue.cc:260] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:23.543586 25201 ts_tablet_manager.cc:1398] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:19:23.544981 13142 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-e141ca70856946a1a3f8e21b77548736 received from {username='slave'} at 127.6.5.2:36987
I20240502 08:19:23.545126 13142 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-e141ca70856946a1a3f8e21b77548736 on tablet e141ca70856946a1a3f8e21b77548736 with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:23.547595 25250 tablet_bootstrap.cc:492] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.547930 25250 tablet_bootstrap.cc:492] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4: Bootstrap complete.
I20240502 08:19:23.548195 25250 ts_tablet_manager.cc:1367] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20240502 08:19:23.548877 25250 raft_consensus.cc:348] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:23.549197 25250 raft_consensus.cc:729] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:19:23.549659 25250 consensus_queue.cc:260] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:23.550421 25250 ts_tablet_manager.cc:1398] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:19:23.553802 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 60b6e265e7754b7187307d05ef405dd4-4ae7246870f14d7da744a1856430487c received from {username='slave'} at 127.6.5.3:52517
I20240502 08:19:23.553951 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 60b6e265e7754b7187307d05ef405dd4-4ae7246870f14d7da744a1856430487c on tablet 4ae7246870f14d7da744a1856430487c with peer 60b6e265e7754b7187307d05ef405dd4
I20240502 08:19:23.569048 12727 consensus_queue.cc:237] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:23.571462 13120 raft_consensus.cc:1261] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:23.571580 12991 raft_consensus.cc:1261] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:23.571982 25166 consensus_queue.cc:1035] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:23.573031 25186 ts_tablet_manager.cc:902] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
W20240502 08:19:23.573662 12664 consensus_peers.cc:485] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee -> Peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Couldn't send request to peer db7893bb29a047b3b82cc60bfd8930a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e249007ce4284fd2bc026ab9d6c5d4cb. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:23.573969 25166 consensus_queue.cc:1035] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:23.575107 25186 tablet_copy_client.cc:308] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:19:23.575680 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet c9ad7bf820c84aae9a8875fb92f8e5d5 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:47559)
I20240502 08:19:23.575831 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet c9ad7bf820c84aae9a8875fb92f8e5d5 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:47559: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-c9ad7bf820c84aae9a8875fb92f8e5d5
I20240502 08:19:23.578255 15508 tablet_copy_source_session.cc:205] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.579485 25186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ad7bf820c84aae9a8875fb92f8e5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.584789 25186 tablet_copy_client.cc:783] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.585176 25186 tablet_copy_client.cc:647] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.586591 25186 tablet_copy_client.cc:523] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.588833 25186 tablet_bootstrap.cc:492] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:19:23.593572 25105 raft_consensus.cc:2932] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:23.596987 13122 raft_consensus.cc:2932] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:23.600657 25186 tablet_bootstrap.cc:492] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.600381 12991 raft_consensus.cc:2932] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:23.601006 25186 tablet_bootstrap.cc:492] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:19:23.601243 25186 ts_tablet_manager.cc:1367] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240502 08:19:23.601908 25186 raft_consensus.cc:348] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:23.602126 25186 raft_consensus.cc:729] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:19:23.602368 25186 consensus_queue.cc:260] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:23.603236 25186 ts_tablet_manager.cc:1398] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:19:23.604238 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-c9ad7bf820c84aae9a8875fb92f8e5d5 received from {username='slave'} at 127.6.5.6:47559
I20240502 08:19:23.604357 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-c9ad7bf820c84aae9a8875fb92f8e5d5 on tablet c9ad7bf820c84aae9a8875fb92f8e5d5 with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:23.608774 12582 catalog_manager.cc:5484] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 4, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:23.653295 15488 consensus_queue.cc:237] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:23.655836 15622 raft_consensus.cc:1261] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:23.656051 12991 raft_consensus.cc:1261] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:23.656541 25145 consensus_queue.cc:1035] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:23.656810 25146 consensus_queue.cc:1035] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:19:23.657089 15419 consensus_peers.cc:485] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Couldn't send request to peer 29b035434087435cbaf4ebcac98eedee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5205c45461da48369a6f0d103f8a94e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:23.660480 25145 raft_consensus.cc:2932] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:23.662873 12991 raft_consensus.cc:2932] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:23.671321 12582 catalog_manager.cc:5484] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:23.677601 15621 raft_consensus.cc:2932] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:23.706125 25310 ts_tablet_manager.cc:902] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:23.709470 25310 tablet_copy_client.cc:308] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:19:23.710201 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet 9366b55ca7dc48bfa5d2ed650d7a30e8 from peer db7893bb29a047b3b82cc60bfd8930a3 ({username='slave'} at 127.6.5.5:46815)
I20240502 08:19:23.710368 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet 9366b55ca7dc48bfa5d2ed650d7a30e8 from peer db7893bb29a047b3b82cc60bfd8930a3 at {username='slave'} at 127.6.5.5:46815: session id = db7893bb29a047b3b82cc60bfd8930a3-9366b55ca7dc48bfa5d2ed650d7a30e8
I20240502 08:19:23.712914 12751 tablet_copy_source_session.cc:205] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.714318 25310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9366b55ca7dc48bfa5d2ed650d7a30e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.720238 25310 tablet_copy_client.cc:783] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.720525 25310 tablet_copy_client.cc:647] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.722240 25310 tablet_copy_client.cc:523] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.725232 25310 tablet_bootstrap.cc:492] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:19:23.732610 15622 consensus_queue.cc:237] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:23.734822 15754 raft_consensus.cc:1261] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:23.735296 15555 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b -> Peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Couldn't send request to peer 10f0e36203914b85b77e78609ffced98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 188e9a523fee46a08a4c6180058ed923. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:23.735481 25185 consensus_queue.cc:1035] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:23.736238 12991 raft_consensus.cc:1261] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:23.736706 25185 consensus_queue.cc:1035] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:23.748745 12860 raft_consensus.cc:1203] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:23.771720 25235 ts_tablet_manager.cc:902] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee: Initiating tablet copy from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:23.772480 25325 ts_tablet_manager.cc:902] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98: Initiating tablet copy from peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:19:23.772876 15622 raft_consensus.cc:1203] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:23.778439 25185 raft_consensus.cc:2932] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:23.783335 12992 raft_consensus.cc:2932] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:23.786828 25310 tablet_bootstrap.cc:492] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.787250 25310 tablet_bootstrap.cc:492] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap complete.
I20240502 08:19:23.787528 25310 ts_tablet_manager.cc:1367] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.062s user 0.059s sys 0.000s
I20240502 08:19:23.788352 25310 raft_consensus.cc:348] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:23.788578 25310 raft_consensus.cc:729] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:19:23.788782 25310 consensus_queue.cc:260] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:23.789419 25310 ts_tablet_manager.cc:1398] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:19:23.790918 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session db7893bb29a047b3b82cc60bfd8930a3-9366b55ca7dc48bfa5d2ed650d7a30e8 received from {username='slave'} at 127.6.5.5:46815
I20240502 08:19:23.791061 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session db7893bb29a047b3b82cc60bfd8930a3-9366b55ca7dc48bfa5d2ed650d7a30e8 on tablet 9366b55ca7dc48bfa5d2ed650d7a30e8 with peer db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:19:23.792271 15755 raft_consensus.cc:2932] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:23.794286 12582 catalog_manager.cc:5484] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 2, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:23.846331 25235 tablet_copy_client.cc:308] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.6:44983
I20240502 08:19:23.847083 15642 tablet_copy_service.cc:140] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Received BeginTabletCopySession request for tablet 10ba9ba78b5444a4bc73be4f6cfd6370 from peer 29b035434087435cbaf4ebcac98eedee ({username='slave'} at 127.6.5.1:42443)
I20240502 08:19:23.847266 15642 tablet_copy_service.cc:161] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Beginning new tablet copy session on tablet 10ba9ba78b5444a4bc73be4f6cfd6370 from peer 29b035434087435cbaf4ebcac98eedee at {username='slave'} at 127.6.5.1:42443: session id = 29b035434087435cbaf4ebcac98eedee-10ba9ba78b5444a4bc73be4f6cfd6370
I20240502 08:19:23.849371 15642 tablet_copy_source_session.cc:205] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.849992 25325 tablet_copy_client.cc:308] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:36933
I20240502 08:19:23.850677 12882 tablet_copy_service.cc:140] P e1dd14f7ca764a1490df419f13f2804a: Received BeginTabletCopySession request for tablet 21f25468f1b541dc9ede27359acc2b9c from peer 10f0e36203914b85b77e78609ffced98 ({username='slave'} at 127.6.5.8:37245)
I20240502 08:19:23.850821 12882 tablet_copy_service.cc:161] P e1dd14f7ca764a1490df419f13f2804a: Beginning new tablet copy session on tablet 21f25468f1b541dc9ede27359acc2b9c from peer 10f0e36203914b85b77e78609ffced98 at {username='slave'} at 127.6.5.8:37245: session id = 10f0e36203914b85b77e78609ffced98-21f25468f1b541dc9ede27359acc2b9c
I20240502 08:19:23.852397 25235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ba9ba78b5444a4bc73be4f6cfd6370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.853380 12882 tablet_copy_source_session.cc:205] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.857038 25325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21f25468f1b541dc9ede27359acc2b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.862053 25235 tablet_copy_client.cc:783] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.862321 25235 tablet_copy_client.cc:647] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.863828 25235 tablet_copy_client.cc:523] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.864993 25325 tablet_copy_client.cc:783] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.865217 25325 tablet_copy_client.cc:647] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.868095 25235 tablet_bootstrap.cc:492] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:19:23.869520 25325 tablet_copy_client.cc:523] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.873594 25325 tablet_bootstrap.cc:492] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:19:23.892424 25235 tablet_bootstrap.cc:492] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.892779 25235 tablet_bootstrap.cc:492] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee: Bootstrap complete.
I20240502 08:19:23.893039 25235 ts_tablet_manager.cc:1367] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.012s
I20240502 08:19:23.893761 25235 raft_consensus.cc:348] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:23.894009 25235 raft_consensus.cc:729] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:19:23.894256 25235 consensus_queue.cc:260] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:23.894937 25235 ts_tablet_manager.cc:1398] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:19:23.895771 15642 tablet_copy_service.cc:342] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Request end of tablet copy session 29b035434087435cbaf4ebcac98eedee-10ba9ba78b5444a4bc73be4f6cfd6370 received from {username='slave'} at 127.6.5.1:42443
I20240502 08:19:23.895956 15642 tablet_copy_service.cc:433] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: ending tablet copy session 29b035434087435cbaf4ebcac98eedee-10ba9ba78b5444a4bc73be4f6cfd6370 on tablet 10ba9ba78b5444a4bc73be4f6cfd6370 with peer 29b035434087435cbaf4ebcac98eedee
I20240502 08:19:23.901084 13122 raft_consensus.cc:1203] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:23.920403 12991 consensus_queue.cc:237] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:23.922370 15755 raft_consensus.cc:1261] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:23.922633 12926 consensus_peers.cc:485] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f50084f1f7784cbfbcb5cb338dbd192e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:23.922911 24609 consensus_queue.cc:1035] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:23.922996 15488 raft_consensus.cc:1261] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:23.923542 24609 consensus_queue.cc:1035] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:23.927553 24276 raft_consensus.cc:2932] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:23.928061 15488 raft_consensus.cc:2932] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:23.929536 25325 tablet_bootstrap.cc:492] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.929879 25325 tablet_bootstrap.cc:492] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98: Bootstrap complete.
I20240502 08:19:23.930089 25325 ts_tablet_manager.cc:1367] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.057s user 0.043s sys 0.002s
I20240502 08:19:23.930748 25325 raft_consensus.cc:348] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:23.930959 25325 raft_consensus.cc:729] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:19:23.931150 25325 consensus_queue.cc:260] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:23.933202 25325 ts_tablet_manager.cc:1398] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 08:19:23.934906 12882 tablet_copy_service.cc:342] P e1dd14f7ca764a1490df419f13f2804a: Request end of tablet copy session 10f0e36203914b85b77e78609ffced98-21f25468f1b541dc9ede27359acc2b9c received from {username='slave'} at 127.6.5.8:37245
I20240502 08:19:23.935027 12882 tablet_copy_service.cc:433] P e1dd14f7ca764a1490df419f13f2804a: ending tablet copy session 10f0e36203914b85b77e78609ffced98-21f25468f1b541dc9ede27359acc2b9c on tablet 21f25468f1b541dc9ede27359acc2b9c with peer 10f0e36203914b85b77e78609ffced98
I20240502 08:19:23.934768 15754 raft_consensus.cc:2932] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:23.941838 12582 catalog_manager.cc:5484] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:23.996510 12727 raft_consensus.cc:1203] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:23.999241 25201 ts_tablet_manager.cc:902] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:19:24.014593 12991 raft_consensus.cc:1203] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:24.015007 25201 tablet_copy_client.cc:308] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:19:24.015791 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet 646aec80c3634b6ba8493832ff4294a0 from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:60127)
I20240502 08:19:24.015938 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet 646aec80c3634b6ba8493832ff4294a0 from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:60127: session id = e1dd14f7ca764a1490df419f13f2804a-646aec80c3634b6ba8493832ff4294a0
I20240502 08:19:24.018720 15508 tablet_copy_source_session.cc:205] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:24.021371 25201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646aec80c3634b6ba8493832ff4294a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:24.022907 25310 ts_tablet_manager.cc:902] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:24.024147 12861 raft_consensus.cc:1203] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:24.027258 25201 tablet_copy_client.cc:783] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:24.027479 25201 tablet_copy_client.cc:647] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:24.029417 25201 tablet_copy_client.cc:523] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:24.033317 15622 raft_consensus.cc:1203] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:24.033878 25201 tablet_bootstrap.cc:492] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:19:24.050035 25201 tablet_bootstrap.cc:492] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:24.050388 25201 tablet_bootstrap.cc:492] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:19:24.050647 25201 ts_tablet_manager.cc:1367] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.005s
I20240502 08:19:24.051321 25201 raft_consensus.cc:348] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:24.051543 25201 raft_consensus.cc:729] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:19:24.051779 25201 consensus_queue.cc:260] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:24.052290 25201 ts_tablet_manager.cc:1398] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:19:24.052824 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-646aec80c3634b6ba8493832ff4294a0 received from {username='slave'} at 127.6.5.2:60127
I20240502 08:19:24.052929 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-646aec80c3634b6ba8493832ff4294a0 on tablet 646aec80c3634b6ba8493832ff4294a0 with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:24.060585 25310 tablet_copy_client.cc:308] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:19:24.061305 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet e249007ce4284fd2bc026ab9d6c5d4cb from peer db7893bb29a047b3b82cc60bfd8930a3 ({username='slave'} at 127.6.5.5:46815)
I20240502 08:19:24.061472 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet e249007ce4284fd2bc026ab9d6c5d4cb from peer db7893bb29a047b3b82cc60bfd8930a3 at {username='slave'} at 127.6.5.5:46815: session id = db7893bb29a047b3b82cc60bfd8930a3-e249007ce4284fd2bc026ab9d6c5d4cb
I20240502 08:19:24.065383 12751 tablet_copy_source_session.cc:205] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:24.067341 25310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e249007ce4284fd2bc026ab9d6c5d4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:24.074564 25310 tablet_copy_client.cc:783] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:24.074769 25310 tablet_copy_client.cc:647] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:24.076212 25310 tablet_copy_client.cc:523] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:24.080005 25310 tablet_bootstrap.cc:492] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:19:24.137735 25325 ts_tablet_manager.cc:902] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98: Initiating tablet copy from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:24.153275 25310 tablet_bootstrap.cc:492] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:24.153591 25310 tablet_bootstrap.cc:492] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap complete.
I20240502 08:19:24.153786 25310 ts_tablet_manager.cc:1367] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.074s user 0.051s sys 0.000s
I20240502 08:19:24.154434 25310 raft_consensus.cc:348] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:24.154750 25310 raft_consensus.cc:729] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:19:24.155030 25310 consensus_queue.cc:260] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:24.155570 25310 ts_tablet_manager.cc:1398] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 08:19:24.156289 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session db7893bb29a047b3b82cc60bfd8930a3-e249007ce4284fd2bc026ab9d6c5d4cb received from {username='slave'} at 127.6.5.5:46815
I20240502 08:19:24.156399 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session db7893bb29a047b3b82cc60bfd8930a3-e249007ce4284fd2bc026ab9d6c5d4cb on tablet e249007ce4284fd2bc026ab9d6c5d4cb with peer db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:19:24.181023 15488 raft_consensus.cc:1203] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:24.186501 25235 ts_tablet_manager.cc:902] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:19:24.187870 25325 tablet_copy_client.cc:308] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.6:44983
I20240502 08:19:24.188539 15642 tablet_copy_service.cc:140] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Received BeginTabletCopySession request for tablet 188e9a523fee46a08a4c6180058ed923 from peer 10f0e36203914b85b77e78609ffced98 ({username='slave'} at 127.6.5.8:46159)
I20240502 08:19:24.188711 15642 tablet_copy_service.cc:161] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Beginning new tablet copy session on tablet 188e9a523fee46a08a4c6180058ed923 from peer 10f0e36203914b85b77e78609ffced98 at {username='slave'} at 127.6.5.8:46159: session id = 10f0e36203914b85b77e78609ffced98-188e9a523fee46a08a4c6180058ed923
I20240502 08:19:24.191715 15642 tablet_copy_source_session.cc:205] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:24.196692 25325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188e9a523fee46a08a4c6180058ed923. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:24.218539 25325 tablet_copy_client.cc:783] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:24.220232 25325 tablet_copy_client.cc:647] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:24.222250 25325 tablet_copy_client.cc:523] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:24.225850 25325 tablet_bootstrap.cc:492] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:19:24.242514 25325 tablet_bootstrap.cc:492] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:24.243374 25325 tablet_bootstrap.cc:492] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98: Bootstrap complete.
I20240502 08:19:24.243759 25325 ts_tablet_manager.cc:1367] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.001s
I20240502 08:19:24.244446 25325 raft_consensus.cc:348] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:24.244812 25325 raft_consensus.cc:729] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:19:24.245074 25325 consensus_queue.cc:260] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:24.245887 25325 ts_tablet_manager.cc:1398] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:19:24.247620 15642 tablet_copy_service.cc:342] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Request end of tablet copy session 10f0e36203914b85b77e78609ffced98-188e9a523fee46a08a4c6180058ed923 received from {username='slave'} at 127.6.5.8:46159
I20240502 08:19:24.247781 15642 tablet_copy_service.cc:433] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: ending tablet copy session 10f0e36203914b85b77e78609ffced98-188e9a523fee46a08a4c6180058ed923 on tablet 188e9a523fee46a08a4c6180058ed923 with peer 10f0e36203914b85b77e78609ffced98
I20240502 08:19:24.248322 15885 raft_consensus.cc:1203] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:24.306056 12727 raft_consensus.cc:1203] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:24.324584 25235 tablet_copy_client.cc:308] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:19:24.325228 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet 5205c45461da48369a6f0d103f8a94e6 from peer 29b035434087435cbaf4ebcac98eedee ({username='slave'} at 127.6.5.1:33495)
I20240502 08:19:24.325398 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet 5205c45461da48369a6f0d103f8a94e6 from peer 29b035434087435cbaf4ebcac98eedee at {username='slave'} at 127.6.5.1:33495: session id = 29b035434087435cbaf4ebcac98eedee-5205c45461da48369a6f0d103f8a94e6
I20240502 08:19:24.328119 15508 tablet_copy_source_session.cc:205] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:24.330446 25235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5205c45461da48369a6f0d103f8a94e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:24.351027 25235 tablet_copy_client.cc:783] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:24.352018 25235 tablet_copy_client.cc:647] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:24.355738 25235 tablet_copy_client.cc:523] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:24.364991 25235 tablet_bootstrap.cc:492] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:19:24.377156 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:24.389798 25235 tablet_bootstrap.cc:492] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:24.390165 25235 tablet_bootstrap.cc:492] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee: Bootstrap complete.
I20240502 08:19:24.390580 25235 ts_tablet_manager.cc:1367] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.010s
I20240502 08:19:24.391342 25235 raft_consensus.cc:348] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:24.391613 25235 raft_consensus.cc:729] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:19:24.391878 25235 consensus_queue.cc:260] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:24.394014 25235 ts_tablet_manager.cc:1398] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:19:24.394711 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session 29b035434087435cbaf4ebcac98eedee-5205c45461da48369a6f0d103f8a94e6 received from {username='slave'} at 127.6.5.1:33495
I20240502 08:19:24.394845 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session 29b035434087435cbaf4ebcac98eedee-5205c45461da48369a6f0d103f8a94e6 on tablet 5205c45461da48369a6f0d103f8a94e6 with peer 29b035434087435cbaf4ebcac98eedee
I20240502 08:19:24.399096 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:24.399215 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:24.400415 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:24.407703 12861 raft_consensus.cc:1203] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:24.408285 25201 ts_tablet_manager.cc:902] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:24.423883 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:19:24.468706 12663 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.7:38851, user_credentials={real_user=slave}} blocked reactor thread for 51550.1us
I20240502 08:19:24.469753 15488 raft_consensus.cc:1203] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:24.508519 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:24.554733 25201 tablet_copy_client.cc:308] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:19:24.555420 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet f50084f1f7784cbfbcb5cb338dbd192e from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:47079)
I20240502 08:19:24.555573 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet f50084f1f7784cbfbcb5cb338dbd192e from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:47079: session id = e1dd14f7ca764a1490df419f13f2804a-f50084f1f7784cbfbcb5cb338dbd192e
I20240502 08:19:24.558691 13012 tablet_copy_source_session.cc:205] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:24.559898 25201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50084f1f7784cbfbcb5cb338dbd192e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:24.561297 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:24.570147 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:24.578541 25201 tablet_copy_client.cc:783] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:24.578792 25201 tablet_copy_client.cc:647] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:24.580435 25201 tablet_copy_client.cc:523] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:24.584221 25201 tablet_bootstrap.cc:492] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:19:24.590219 25201 tablet_bootstrap.cc:492] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:24.590690 25201 tablet_bootstrap.cc:492] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:19:24.590926 25201 ts_tablet_manager.cc:1367] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240502 08:19:24.591642 25201 raft_consensus.cc:348] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:24.591867 25201 raft_consensus.cc:729] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:19:24.592095 25201 consensus_queue.cc:260] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:24.592810 25201 ts_tablet_manager.cc:1398] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:19:24.593497 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-f50084f1f7784cbfbcb5cb338dbd192e received from {username='slave'} at 127.6.5.2:47079
I20240502 08:19:24.593603 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-f50084f1f7784cbfbcb5cb338dbd192e on tablet f50084f1f7784cbfbcb5cb338dbd192e with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:24.689307 15885 raft_consensus.cc:1203] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:24.773005 12727 raft_consensus.cc:1203] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:24.835002 12860 raft_consensus.cc:1203] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:25.021992 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:25.032797 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:25.067205 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:25.092554 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:25.112438 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:25.126358 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:25.128509 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:25.181965 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:25.315485 12989 consensus_queue.cc:237] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:25.317294 12861 raft_consensus.cc:1261] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:25.317387 15488 raft_consensus.cc:1261] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:25.317883 25523 consensus_queue.cc:1035] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:25.318106 24609 consensus_queue.cc:1035] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:25.320734 24609 raft_consensus.cc:2932] T f50084f1f7784cbfbcb5cb338dbd192e P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:25.321832 15488 raft_consensus.cc:2932] T f50084f1f7784cbfbcb5cb338dbd192e P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:25.321743 12861 raft_consensus.cc:2932] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:25.328279 12579 catalog_manager.cc:5484] T f50084f1f7784cbfbcb5cb338dbd192e P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:25.346880 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet f50084f1f7784cbfbcb5cb338dbd192e with delete_type TABLET_DATA_TOMBSTONED (TS 9333cb10227b4801b4fa7c24e22e131b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45134
I20240502 08:19:25.357605 25551 tablet_replica.cc:330] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:19:25.357802 25551 raft_consensus.cc:2229] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:25.357997 25551 raft_consensus.cc:2258] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:25.358980 25551 ts_tablet_manager.cc:1875] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:25.364192 25551 ts_tablet_manager.cc:1888] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:25.364280 25551 log.cc:1196] T f50084f1f7784cbfbcb5cb338dbd192e P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/f50084f1f7784cbfbcb5cb338dbd192e
I20240502 08:19:25.364892 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet f50084f1f7784cbfbcb5cb338dbd192e (table rebalancer_extra_table_42 [id=2f80953a600947b79f5e500d1cb36f09]) successfully deleted
I20240502 08:19:25.649278 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:25.680608 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:25.685575 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:25.763289 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:25.790998 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:25.816022 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:25.857152 12577 maintenance_manager.cc:419] P 17be716fc99c4d588d7726089a2116d2: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033384
I20240502 08:19:25.891202 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:25.896486 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:25.989259 12576 maintenance_manager.cc:643] P 17be716fc99c4d588d7726089a2116d2: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.131s user 0.062s sys 0.011s Metrics: {"bytes_written":101535,"cfile_init":1,"dirs.queue_time_us":885,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1021,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1361,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":287,"thread_start_us":790,"threads_started":1}
I20240502 08:19:25.990685 12577 maintenance_manager.cc:419] P 17be716fc99c4d588d7726089a2116d2: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 33581 bytes on disk
I20240502 08:19:25.991503 12576 maintenance_manager.cc:643] P 17be716fc99c4d588d7726089a2116d2: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 08:19:26.016690 15484 consensus_queue.cc:237] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:26.018400 12727 raft_consensus.cc:1261] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:26.018594 12989 raft_consensus.cc:1261] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:26.018975 25145 consensus_queue.cc:1035] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:26.019234 25146 consensus_queue.cc:1035] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:26.021836 25146 raft_consensus.cc:2932] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:26.022496 12989 raft_consensus.cc:2932] T 5205c45461da48369a6f0d103f8a94e6 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:26.026084 12727 raft_consensus.cc:2932] T 5205c45461da48369a6f0d103f8a94e6 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:26.027375 12581 catalog_manager.cc:5484] T 5205c45461da48369a6f0d103f8a94e6 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:26.045830 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 5205c45461da48369a6f0d103f8a94e6 with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:26.058857 25639 tablet_replica.cc:330] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:26.059226 25639 raft_consensus.cc:2229] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:26.059564 25639 raft_consensus.cc:2258] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:26.060227 25639 ts_tablet_manager.cc:1875] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:26.067852 25639 ts_tablet_manager.cc:1888] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:26.067945 25639 log.cc:1196] T 5205c45461da48369a6f0d103f8a94e6 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/5205c45461da48369a6f0d103f8a94e6
I20240502 08:19:26.068559 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 5205c45461da48369a6f0d103f8a94e6 (table rebalancer_extra_table_12 [id=69a26fefc8344774a5308e75413de1d7]) successfully deleted
I20240502 08:19:26.287014 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:26.318677 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:26.328312 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:26.337359 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 08:19:26.373584 15689 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.8:46303, user_credentials={real_user=slave}} blocked reactor thread for 45902.5us
I20240502 08:19:26.408071 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:26.420410 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:26.452289 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:26.480958 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:26.577586 12727 consensus_queue.cc:237] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:26.579916 15488 raft_consensus.cc:1261] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:26.580003 12989 raft_consensus.cc:1261] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:26.580534 25439 consensus_queue.cc:1035] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:26.580828 25105 consensus_queue.cc:1035] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:26.583268 25439 raft_consensus.cc:2932] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:26.583705 12989 raft_consensus.cc:2932] T e249007ce4284fd2bc026ab9d6c5d4cb P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:26.583806 15488 raft_consensus.cc:2932] T e249007ce4284fd2bc026ab9d6c5d4cb P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:26.590682 12585 catalog_manager.cc:5484] T e249007ce4284fd2bc026ab9d6c5d4cb P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:26.615743 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet e249007ce4284fd2bc026ab9d6c5d4cb with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:19:26.616531 25729 tablet_replica.cc:330] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:19:26.616768 25729 raft_consensus.cc:2229] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:26.616990 25729 raft_consensus.cc:2258] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:26.618207 25729 ts_tablet_manager.cc:1875] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:26.624512 25729 ts_tablet_manager.cc:1888] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:26.624708 25729 log.cc:1196] T e249007ce4284fd2bc026ab9d6c5d4cb P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/e249007ce4284fd2bc026ab9d6c5d4cb
I20240502 08:19:26.630688 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet e249007ce4284fd2bc026ab9d6c5d4cb (table rebalance_test_table_17 [id=c8304b52e21948eba4b4290948c74b5d]) successfully deleted
I20240502 08:19:26.883190 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:26.891475 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:26.893680 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:26.929028 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:26.937737 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:26.947940 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:26.947968 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:26.979796 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:27.145802 15484 consensus_queue.cc:237] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:27.147835 12860 raft_consensus.cc:1261] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:27.147962 12727 raft_consensus.cc:1261] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:27.148375 25145 consensus_queue.cc:1035] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:27.148640 25146 consensus_queue.cc:1035] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:27.151449 25146 raft_consensus.cc:2932] T 646aec80c3634b6ba8493832ff4294a0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:27.151873 12860 raft_consensus.cc:2932] T 646aec80c3634b6ba8493832ff4294a0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:27.152140 12727 raft_consensus.cc:2932] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:27.159782 12581 catalog_manager.cc:5484] T 646aec80c3634b6ba8493832ff4294a0 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:27.166267 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 646aec80c3634b6ba8493832ff4294a0 with delete_type TABLET_DATA_TOMBSTONED (TS 9333cb10227b4801b4fa7c24e22e131b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45134
I20240502 08:19:27.174786 25799 tablet_replica.cc:330] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:19:27.174973 25799 raft_consensus.cc:2229] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:27.175168 25799 raft_consensus.cc:2258] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:27.175689 25799 ts_tablet_manager.cc:1875] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:27.180585 25799 ts_tablet_manager.cc:1888] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:27.180691 25799 log.cc:1196] T 646aec80c3634b6ba8493832ff4294a0 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/646aec80c3634b6ba8493832ff4294a0
I20240502 08:19:27.181351 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 646aec80c3634b6ba8493832ff4294a0 (table rebalancer_extra_table_6 [id=f19b8479d1b74f17b2ee9e2117749102]) successfully deleted
I20240502 08:19:27.481375 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:27.527254 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:27.541743 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:27.555577 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:27.575481 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:27.587797 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:27.600796 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:27.615134 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:27.783538 12861 consensus_queue.cc:237] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:27.786149 15887 raft_consensus.cc:1261] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:27.786378 12730 raft_consensus.cc:1261] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:27.786801 25350 consensus_queue.cc:1035] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:27.787264 25760 consensus_queue.cc:1035] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:27.795728 25874 raft_consensus.cc:2932] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:27.798689 12730 raft_consensus.cc:2932] T 21f25468f1b541dc9ede27359acc2b9c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:27.803928 15887 raft_consensus.cc:2932] T 21f25468f1b541dc9ede27359acc2b9c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:27.805100 12588 catalog_manager.cc:5484] T 21f25468f1b541dc9ede27359acc2b9c P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index 4 to 5, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:27.845945 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 21f25468f1b541dc9ede27359acc2b9c with delete_type TABLET_DATA_TOMBSTONED (TS 60b6e265e7754b7187307d05ef405dd4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38740
I20240502 08:19:27.846455 25888 tablet_replica.cc:330] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:19:27.847399 25888 raft_consensus.cc:2229] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:27.847584 25888 raft_consensus.cc:2258] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:27.848246 25888 ts_tablet_manager.cc:1875] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:27.855269 25888 ts_tablet_manager.cc:1888] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:27.855372 25888 log.cc:1196] T 21f25468f1b541dc9ede27359acc2b9c P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/21f25468f1b541dc9ede27359acc2b9c
I20240502 08:19:27.856139 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 21f25468f1b541dc9ede27359acc2b9c (table rebalance_test_table_3 [id=f32118c718584379bfc02c6d1b29d73e]) successfully deleted
I20240502 08:19:28.064847 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:28.102579 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:28.110040 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:28.110520 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:28.136729 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:28.180258 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:28.184713 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:28.262804 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:28.375161 12727 consensus_queue.cc:237] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:28.377285 15488 raft_consensus.cc:1261] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:28.377312 12991 raft_consensus.cc:1261] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:28.377941 25439 consensus_queue.cc:1035] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:28.378238 25714 consensus_queue.cc:1035] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:28.383306 25714 raft_consensus.cc:2932] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:28.383630 12991 raft_consensus.cc:2932] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:28.389709 15484 raft_consensus.cc:2932] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:28.390980 12588 catalog_manager.cc:5484] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:28.413007 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 9366b55ca7dc48bfa5d2ed650d7a30e8 with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:19:28.413606 25982 tablet_replica.cc:330] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:19:28.413796 25982 raft_consensus.cc:2229] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:28.414772 25982 raft_consensus.cc:2258] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:28.418207 25982 ts_tablet_manager.cc:1875] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:28.425050 25982 ts_tablet_manager.cc:1888] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:28.425189 25982 log.cc:1196] T 9366b55ca7dc48bfa5d2ed650d7a30e8 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/9366b55ca7dc48bfa5d2ed650d7a30e8
I20240502 08:19:28.425853 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 9366b55ca7dc48bfa5d2ed650d7a30e8 (table rebalance_test_table_7 [id=7ed6dd5b8e8d45edb609b77e58525533]) successfully deleted
I20240502 08:19:28.692524 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:28.697604 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:28.717617 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:28.727622 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:28.803429 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:28.813838 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:28.854024 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:28.887048 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:29.004110 12727 consensus_queue.cc:237] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:29.006119 15885 raft_consensus.cc:1261] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:29.006119 12989 raft_consensus.cc:1261] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:29.006678 25714 consensus_queue.cc:1035] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:29.007007 25439 consensus_queue.cc:1035] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:29.010135 26050 raft_consensus.cc:2932] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:29.011261 12989 raft_consensus.cc:2932] T 4ae7246870f14d7da744a1856430487c P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:29.016294 15887 raft_consensus.cc:2932] T 4ae7246870f14d7da744a1856430487c P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:29.020498 12588 catalog_manager.cc:5484] T 4ae7246870f14d7da744a1856430487c P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:29.075271 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae7246870f14d7da744a1856430487c with delete_type TABLET_DATA_TOMBSTONED (TS db7893bb29a047b3b82cc60bfd8930a3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46880
I20240502 08:19:29.081668 26068 tablet_replica.cc:330] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:19:29.081949 26068 raft_consensus.cc:2229] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:29.082198 26068 raft_consensus.cc:2258] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:29.083043 26068 ts_tablet_manager.cc:1875] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:29.093955 26068 ts_tablet_manager.cc:1888] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:29.094048 26068 log.cc:1196] T 4ae7246870f14d7da744a1856430487c P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/4ae7246870f14d7da744a1856430487c
I20240502 08:19:29.094708 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 4ae7246870f14d7da744a1856430487c (table rebalancer_extra_table_30 [id=356db21cc0814b908e5333ba8c2e04ea]) successfully deleted
I20240502 08:19:29.271085 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:29.292126 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:29.300166 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:29.300431 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:29.379237 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:29.431156 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:29.463429 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:29.474303 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:29.580035 12860 consensus_queue.cc:237] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:29.582098 15754 raft_consensus.cc:1261] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:29.582309 12727 raft_consensus.cc:1261] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:29.582684 25874 consensus_queue.cc:1035] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:29.583344 26109 consensus_queue.cc:1035] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:29.585850 25874 raft_consensus.cc:2932] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:29.586604 12727 raft_consensus.cc:2932] T c2ad585a32864067a5d8eaea5f8260f1 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:29.597301 12588 catalog_manager.cc:5484] T c2ad585a32864067a5d8eaea5f8260f1 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:29.597461 15754 raft_consensus.cc:2932] T c2ad585a32864067a5d8eaea5f8260f1 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:29.616055 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ad585a32864067a5d8eaea5f8260f1 with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:29.616657 26140 tablet_replica.cc:330] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:29.616899 26140 raft_consensus.cc:2229] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:29.617120 26140 raft_consensus.cc:2258] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:29.617753 26140 ts_tablet_manager.cc:1875] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:29.626937 26140 ts_tablet_manager.cc:1888] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:29.627140 26140 log.cc:1196] T c2ad585a32864067a5d8eaea5f8260f1 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/c2ad585a32864067a5d8eaea5f8260f1
I20240502 08:19:29.628106 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet c2ad585a32864067a5d8eaea5f8260f1 (table rebalancer_extra_table_10 [id=e77588602bd34d11a274be948cc9d52a]) successfully deleted
I20240502 08:19:29.897692 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:29.953109 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:29.957787 12708 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240502 08:19:29.990643 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:29.990993 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:30.000339 15600 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:30.010661 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:30.032958 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:30.177987 15622 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10ba9ba78b5444a4bc73be4f6cfd6370"
dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38668
I20240502 08:19:30.178270 15622 raft_consensus.cc:595] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:30.297962 26213 raft_consensus.cc:979] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: : Instructing follower 29b035434087435cbaf4ebcac98eedee to start an election
I20240502 08:19:30.298108 26213 raft_consensus.cc:1067] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Signalling peer 29b035434087435cbaf4ebcac98eedee to start an election
I20240502 08:19:30.299405 12727 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10ba9ba78b5444a4bc73be4f6cfd6370"
dest_uuid: "29b035434087435cbaf4ebcac98eedee"
from {username='slave'} at 127.6.5.6:55577
I20240502 08:19:30.299604 12727 raft_consensus.cc:482] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:30.299692 12727 raft_consensus.cc:3037] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:30.302877 12727 raft_consensus.cc:504] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:30.304240 15622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ba9ba78b5444a4bc73be4f6cfd6370" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:19:30.304425 15622 raft_consensus.cc:3032] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:30.304503 15622 raft_consensus.cc:729] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:19:30.304759 15622 consensus_queue.cc:260] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:30.305136 15622 raft_consensus.cc:3037] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:30.307595 15622 raft_consensus.cc:2454] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 2.
I20240502 08:19:30.308846 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ba9ba78b5444a4bc73be4f6cfd6370" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:19:30.308950 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ba9ba78b5444a4bc73be4f6cfd6370" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:19:30.309047 15754 raft_consensus.cc:3037] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:30.309114 15887 raft_consensus.cc:3037] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:30.310511 12727 leader_election.cc:288] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:19:30.311659 15887 raft_consensus.cc:2454] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 2.
I20240502 08:19:30.312067 12665 leader_election.cc:302] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 29b035434087435cbaf4ebcac98eedee, 8d45ab45b8eb4d03a4bb9cb88b78a98b; no voters:
I20240502 08:19:30.312156 15754 raft_consensus.cc:2454] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 2.
I20240502 08:19:30.312352 26050 raft_consensus.cc:2781] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:30.312479 26050 raft_consensus.cc:686] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [term 2 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:19:30.312709 26050 consensus_queue.cc:237] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:30.317589 12587 catalog_manager.cc:5484] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 2 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:30.470805 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:30.556445 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:30.561280 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:30.563480 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:30.596534 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:30.650669 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:30.651324 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:30.667634 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:30.695421 15622 raft_consensus.cc:1261] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 2 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:19:30.696009 26271 consensus_queue.cc:1035] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:30.717693 15754 raft_consensus.cc:1261] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:19:30.718402 26271 consensus_queue.cc:1035] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:30.758319 15887 raft_consensus.cc:1261] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:19:30.759270 26193 consensus_queue.cc:1035] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:31.100641 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:31.101089 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:31.105010 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:31.133877 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:31.151572 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:31.156492 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:31.194736 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:31.217942 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:31.329382 12991 consensus_queue.cc:237] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:31.331202 15488 raft_consensus.cc:1261] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:31.331202 13122 raft_consensus.cc:1261] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:31.331738 24609 consensus_queue.cc:1035] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:31.332036 26194 consensus_queue.cc:1035] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:31.334692 24609 raft_consensus.cc:2932] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:31.335294 15488 raft_consensus.cc:2932] T 1d3592822ca849b9a47c4320b84754ae P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:31.341038 12586 catalog_manager.cc:5484] T 1d3592822ca849b9a47c4320b84754ae P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:31.343815 13121 raft_consensus.cc:2932] T 1d3592822ca849b9a47c4320b84754ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:31.370399 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3592822ca849b9a47c4320b84754ae with delete_type TABLET_DATA_TOMBSTONED (TS 10f0e36203914b85b77e78609ffced98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47272
I20240502 08:19:31.378810 26376 tablet_replica.cc:330] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:19:31.379009 26376 raft_consensus.cc:2229] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:31.379197 26376 raft_consensus.cc:2258] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:31.379786 26376 ts_tablet_manager.cc:1875] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:31.386336 26376 ts_tablet_manager.cc:1888] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:31.386560 26376 log.cc:1196] T 1d3592822ca849b9a47c4320b84754ae P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/1d3592822ca849b9a47c4320b84754ae
I20240502 08:19:31.387250 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 1d3592822ca849b9a47c4320b84754ae (table rebalancer_extra_table_18 [id=320abaf9751a4925a5200feff7f4eacc]) successfully deleted
I20240502 08:19:31.659988 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:31.686738 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:31.711410 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:31.726011 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:31.785063 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:31.838158 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:31.872642 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:31.875702 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:31.980286 13121 consensus_queue.cc:237] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:31.982652 12860 raft_consensus.cc:1261] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:31.982722 15487 raft_consensus.cc:1261] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:31.983104 25329 consensus_queue.cc:1035] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:31.983381 26339 consensus_queue.cc:1035] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:31.986660 25329 raft_consensus.cc:2932] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:31.987704 12860 raft_consensus.cc:2932] T e141ca70856946a1a3f8e21b77548736 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:31.996140 12588 catalog_manager.cc:5484] T e141ca70856946a1a3f8e21b77548736 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:32.009505 15487 raft_consensus.cc:2932] T e141ca70856946a1a3f8e21b77548736 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:32.021703 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet e141ca70856946a1a3f8e21b77548736 with delete_type TABLET_DATA_TOMBSTONED (TS 9333cb10227b4801b4fa7c24e22e131b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45134
I20240502 08:19:32.022809 26456 tablet_replica.cc:330] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:19:32.023013 26456 raft_consensus.cc:2229] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:32.023249 26456 raft_consensus.cc:2258] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:32.023917 26456 ts_tablet_manager.cc:1875] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:32.028733 26456 ts_tablet_manager.cc:1888] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:32.028856 26456 log.cc:1196] T e141ca70856946a1a3f8e21b77548736 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/e141ca70856946a1a3f8e21b77548736
I20240502 08:19:32.029660 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet e141ca70856946a1a3f8e21b77548736 (table rebalancer_extra_table_56 [id=97336a3aa550477cb0f32e30f0e2a47c]) successfully deleted
I20240502 08:19:32.274528 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:32.302397 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:32.320525 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:32.320890 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:32.347882 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:32.353286 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:32.370481 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:32.406189 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:32.562348 15621 consensus_queue.cc:237] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:32.564630 12861 raft_consensus.cc:1261] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:32.565088 15887 raft_consensus.cc:1261] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:32.565093 26512 consensus_queue.cc:1035] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:32.565552 26395 consensus_queue.cc:1035] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:32.568607 26512 raft_consensus.cc:2932] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:32.570909 12861 raft_consensus.cc:2932] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:32.574368 12585 catalog_manager.cc:5484] T 8de012363e4f4455acd96cb674ab0c4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:32.598894 15885 raft_consensus.cc:2932] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:32.600956 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de012363e4f4455acd96cb674ab0c4b with delete_type TABLET_DATA_TOMBSTONED (TS 9333cb10227b4801b4fa7c24e22e131b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45134
I20240502 08:19:32.605095 26535 tablet_replica.cc:330] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:19:32.606158 26535 raft_consensus.cc:2229] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:32.606401 26535 raft_consensus.cc:2258] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:32.607198 26535 ts_tablet_manager.cc:1875] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:32.612699 26535 ts_tablet_manager.cc:1888] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:32.612787 26535 log.cc:1196] T 8de012363e4f4455acd96cb674ab0c4b P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/8de012363e4f4455acd96cb674ab0c4b
I20240502 08:19:32.613471 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 8de012363e4f4455acd96cb674ab0c4b (table rebalancer_extra_table_44 [id=6eff13bc8a8247babea96b8f0537a4cf]) successfully deleted
I20240502 08:19:32.927392 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:32.938623 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:32.944265 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:32.986936 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:33.035277 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:33.065623 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:33.074199 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:33.107225 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:33.205217 15487 consensus_queue.cc:237] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:33.207209 15622 raft_consensus.cc:1261] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:33.207352 12730 raft_consensus.cc:1261] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:33.207937 26421 consensus_queue.cc:1035] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:33.208336 26421 consensus_queue.cc:1035] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:33.210829 26279 raft_consensus.cc:2932] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:33.212530 15622 raft_consensus.cc:2932] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:33.217057 12730 raft_consensus.cc:2932] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:33.218964 12580 catalog_manager.cc:5484] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:33.245054 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet c9ad7bf820c84aae9a8875fb92f8e5d5 with delete_type TABLET_DATA_TOMBSTONED (TS 10f0e36203914b85b77e78609ffced98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47272
I20240502 08:19:33.246335 26634 tablet_replica.cc:330] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:19:33.246649 26634 raft_consensus.cc:2229] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:33.246999 26634 raft_consensus.cc:2258] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:33.247776 26634 ts_tablet_manager.cc:1875] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:33.254233 26634 ts_tablet_manager.cc:1888] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:33.254417 26634 log.cc:1196] T c9ad7bf820c84aae9a8875fb92f8e5d5 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/c9ad7bf820c84aae9a8875fb92f8e5d5
I20240502 08:19:33.256165 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet c9ad7bf820c84aae9a8875fb92f8e5d5 (table rebalancer_extra_table_46 [id=7b4c833a5fb54b3e967ac5b77b0558d3]) successfully deleted
I20240502 08:19:33.577746 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:33.588268 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:33.604126 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:33.644981 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:33.658691 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:33.665693 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:33.669627 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:33.669836 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:33.870908 12991 consensus_queue.cc:237] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:33.872665 15619 raft_consensus.cc:1261] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:33.872726 13122 raft_consensus.cc:1261] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:33.873189 24609 consensus_queue.cc:1035] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:33.873458 26670 consensus_queue.cc:1035] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:33.876827 26670 raft_consensus.cc:2932] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:33.878288 13122 raft_consensus.cc:2932] T 553cec4eb28341dba84afc79b540ae4b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:33.879285 15619 raft_consensus.cc:2932] T 553cec4eb28341dba84afc79b540ae4b P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:33.886925 12582 catalog_manager.cc:5484] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 4 to 5, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:33.892413 12708 tablet_service.cc:1514] Processing DeleteTablet for tablet 553cec4eb28341dba84afc79b540ae4b with delete_type TABLET_DATA_TOMBSTONED (TS 29b035434087435cbaf4ebcac98eedee not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40606
I20240502 08:19:33.892982 26733 tablet_replica.cc:330] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee: stopping tablet replica
I20240502 08:19:33.893180 26733 raft_consensus.cc:2229] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:33.893352 26733 raft_consensus.cc:2258] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:33.894146 26733 ts_tablet_manager.cc:1875] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:33.900215 26733 ts_tablet_manager.cc:1888] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:33.900321 26733 log.cc:1196] T 553cec4eb28341dba84afc79b540ae4b P 29b035434087435cbaf4ebcac98eedee: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-0/wal/wals/553cec4eb28341dba84afc79b540ae4b
I20240502 08:19:33.900871 12573 catalog_manager.cc:4830] TS 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): tablet 553cec4eb28341dba84afc79b540ae4b (table rebalance_test_table_17 [id=c8304b52e21948eba4b4290948c74b5d]) successfully deleted
I20240502 08:19:34.142843 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:34.143249 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:34.158349 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:34.169389 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:34.230137 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:34.236258 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:34.239845 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:34.272356 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:34.411867 12730 consensus_queue.cc:237] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:34.414136 12862 raft_consensus.cc:1261] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:34.414350 15755 raft_consensus.cc:1261] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:34.414698 26624 consensus_queue.cc:1035] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:34.415575 26624 consensus_queue.cc:1035] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:34.418028 26050 raft_consensus.cc:2932] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:34.421612 12862 raft_consensus.cc:2932] T c24538742b874c92a70e17b3450993fb P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:34.421617 15755 raft_consensus.cc:2932] T c24538742b874c92a70e17b3450993fb P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:34.432507 12582 catalog_manager.cc:5484] T c24538742b874c92a70e17b3450993fb P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:34.452368 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(ea9c28b7e3b1456d96330665460a93ac): perf score=0.033338
I20240502 08:19:34.453869 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet c24538742b874c92a70e17b3450993fb with delete_type TABLET_DATA_TOMBSTONED (TS 10f0e36203914b85b77e78609ffced98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47272
I20240502 08:19:34.454411 26832 tablet_replica.cc:330] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:19:34.454699 26832 raft_consensus.cc:2229] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:34.454946 26832 raft_consensus.cc:2258] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:34.455675 26832 ts_tablet_manager.cc:1875] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:34.462208 26832 ts_tablet_manager.cc:1888] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:34.462371 26832 log.cc:1196] T c24538742b874c92a70e17b3450993fb P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/c24538742b874c92a70e17b3450993fb
I20240502 08:19:34.463205 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet c24538742b874c92a70e17b3450993fb (table rebalancer_extra_table_40 [id=bdfb8acace12439cb1fe7b02ffd810cb]) successfully deleted
I20240502 08:19:34.530117 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(ea9c28b7e3b1456d96330665460a93ac): perf score=0.033359
I20240502 08:19:34.573961 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(83e0c78cb2374ba483315ca8a827e532): perf score=0.033366
I20240502 08:19:34.652686 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(1f07b2caf1754c469a8e33382f615019): perf score=0.033394
I20240502 08:19:34.711781 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(ea9c28b7e3b1456d96330665460a93ac) complete. Timing: real 0.259s user 0.068s sys 0.034s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":16823,"dirs.queue_time_us":6306,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":13537,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3103,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":48665,"threads_started":3,"wal-append.queue_time_us":30559}
I20240502 08:19:34.717403 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(ea9c28b7e3b1456d96330665460a93ac): 4586 bytes on disk
I20240502 08:19:34.718354 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(ea9c28b7e3b1456d96330665460a93ac) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 08:19:34.721571 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(83e0c78cb2374ba483315ca8a827e532): perf score=0.033408
I20240502 08:19:34.785293 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(ea9c28b7e3b1456d96330665460a93ac) complete. Timing: real 0.254s user 0.062s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":17610,"dirs.queue_time_us":26542,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":6089,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":803,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":74640,"threads_started":3,"wal-append.queue_time_us":16573}
I20240502 08:19:34.799700 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(1f07b2caf1754c469a8e33382f615019): perf score=0.033429
I20240502 08:19:34.843793 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(83e0c78cb2374ba483315ca8a827e532) complete. Timing: real 0.269s user 0.070s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":17004,"dirs.queue_time_us":9714,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":18049,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11950,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":689,"thread_start_us":42940,"threads_started":3,"wal-append.queue_time_us":15496}
I20240502 08:19:34.848345 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(83e0c78cb2374ba483315ca8a827e532): 4925 bytes on disk
I20240502 08:19:34.853248 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(83e0c78cb2374ba483315ca8a827e532) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 08:19:34.855778 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(ea9c28b7e3b1456d96330665460a93ac): perf score=0.033440
I20240502 08:19:34.868067 13121 consensus_queue.cc:237] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
W20240502 08:19:34.898777 13056 consensus_peers.cc:485] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): Couldn't send request to peer 60b6e265e7754b7187307d05ef405dd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51fe8c8f38d4492194f6e3c83f25a60b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:34.899437 15487 raft_consensus.cc:1261] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:34.906692 26451 consensus_queue.cc:1035] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:34.923801 15887 raft_consensus.cc:1261] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:34.924463 26451 consensus_queue.cc:1035] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:34.953652 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(83e0c78cb2374ba483315ca8a827e532) complete. Timing: real 0.232s user 0.052s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1720,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1933,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":36448,"threads_started":1,"wal-append.queue_time_us":38263}
I20240502 08:19:34.957252 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(83e0c78cb2374ba483315ca8a827e532): 4925 bytes on disk
I20240502 08:19:34.957885 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(83e0c78cb2374ba483315ca8a827e532) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 08:19:34.961568 26339 raft_consensus.cc:2932] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:34.966172 15885 raft_consensus.cc:2932] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:34.978785 12585 catalog_manager.cc:5484] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:34.981616 15487 raft_consensus.cc:2932] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:34.984800 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(ea9c28b7e3b1456d96330665460a93ac) complete. Timing: real 0.128s user 0.035s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4447,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1280,"drs_written":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4,"lbm_write_time_us":745,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6496,"threads_started":1,"wal-append.queue_time_us":6412}
I20240502 08:19:34.987658 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(ea9c28b7e3b1456d96330665460a93ac): 4586 bytes on disk
I20240502 08:19:34.988250 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(ea9c28b7e3b1456d96330665460a93ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 08:19:35.060011 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(1f07b2caf1754c469a8e33382f615019) complete. Timing: real 0.407s user 0.064s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":26054,"dirs.queue_time_us":20317,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":3813,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":26708,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":71453,"threads_started":3,"wal-append.queue_time_us":9829}
I20240502 08:19:35.068662 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(1f07b2caf1754c469a8e33382f615019): 4838 bytes on disk
I20240502 08:19:35.070067 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(1f07b2caf1754c469a8e33382f615019) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240502 08:19:35.098691 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(1f07b2caf1754c469a8e33382f615019) complete. Timing: real 0.292s user 0.048s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":12864,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1299,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":836,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":15272,"threads_started":1,"wal-append.queue_time_us":15199}
I20240502 08:19:35.102340 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(1f07b2caf1754c469a8e33382f615019): 4838 bytes on disk
I20240502 08:19:35.105249 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(1f07b2caf1754c469a8e33382f615019) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 08:19:35.107758 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(ea9c28b7e3b1456d96330665460a93ac): 4586 bytes on disk
I20240502 08:19:35.114969 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(ea9c28b7e3b1456d96330665460a93ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240502 08:19:35.253113 15755 consensus_queue.cc:237] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:35.264086 12861 raft_consensus.cc:1261] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:35.264104 15687 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Couldn't send request to peer 6a0b05f3250d415590b0edf6b44fbe1f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 815d79a2bbba49ea9e135d94c57a3390. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:35.264642 25853 consensus_queue.cc:1035] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:35.271924 15622 raft_consensus.cc:1261] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:35.272423 25853 consensus_queue.cc:1035] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:35.356210 26898 raft_consensus.cc:2932] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:35.361413 15622 raft_consensus.cc:2932] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
W20240502 08:19:35.363119 15688 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.5:44131, user_credentials={real_user=slave}} blocked reactor thread for 49351.3us
I20240502 08:19:35.377377 12584 catalog_manager.cc:5484] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:35.382731 12862 raft_consensus.cc:2932] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:35.420025 26904 ts_tablet_manager.cc:902] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:19:35.495661 26904 tablet_copy_client.cc:308] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:19:35.497481 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet 51fe8c8f38d4492194f6e3c83f25a60b from peer 60b6e265e7754b7187307d05ef405dd4 ({username='slave'} at 127.6.5.3:46755)
I20240502 08:19:35.497617 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet 51fe8c8f38d4492194f6e3c83f25a60b from peer 60b6e265e7754b7187307d05ef405dd4 at {username='slave'} at 127.6.5.3:46755: session id = 60b6e265e7754b7187307d05ef405dd4-51fe8c8f38d4492194f6e3c83f25a60b
I20240502 08:19:35.548281 13142 tablet_copy_source_session.cc:205] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:35.552706 26904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fe8c8f38d4492194f6e3c83f25a60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:35.560668 26904 tablet_copy_client.cc:783] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:35.563053 26904 tablet_copy_client.cc:647] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:35.564867 26904 tablet_copy_client.cc:523] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:35.568084 26904 tablet_bootstrap.cc:492] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4: Bootstrap starting.
I20240502 08:19:35.598381 26904 tablet_bootstrap.cc:492] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:35.598946 26904 tablet_bootstrap.cc:492] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4: Bootstrap complete.
I20240502 08:19:35.599324 26904 ts_tablet_manager.cc:1367] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.000s
I20240502 08:19:35.600313 26904 raft_consensus.cc:348] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:35.602535 26904 raft_consensus.cc:729] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Initialized, Role: FOLLOWER
I20240502 08:19:35.602838 26904 consensus_queue.cc:260] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:35.605701 26904 ts_tablet_manager.cc:1398] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:19:35.606966 13142 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session 60b6e265e7754b7187307d05ef405dd4-51fe8c8f38d4492194f6e3c83f25a60b received from {username='slave'} at 127.6.5.3:46755
I20240502 08:19:35.607071 13142 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session 60b6e265e7754b7187307d05ef405dd4-51fe8c8f38d4492194f6e3c83f25a60b on tablet 51fe8c8f38d4492194f6e3c83f25a60b with peer 60b6e265e7754b7187307d05ef405dd4
I20240502 08:19:35.647348 15619 consensus_queue.cc:237] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
W20240502 08:19:35.663570 15554 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b -> Peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): Couldn't send request to peer db7893bb29a047b3b82cc60bfd8930a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d4a5f0c5c184b7f975f7bcc59cb6b4c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:35.664445 13122 raft_consensus.cc:1261] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:35.664927 12727 raft_consensus.cc:1261] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:35.665422 26395 consensus_queue.cc:1035] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:35.666249 26395 consensus_queue.cc:1035] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:35.688951 26395 raft_consensus.cc:2932] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:35.692876 13121 raft_consensus.cc:2932] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:35.699712 12582 catalog_manager.cc:5484] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index -1 to 2, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:35.701539 12727 raft_consensus.cc:2932] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:35.779047 26931 ts_tablet_manager.cc:902] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f: Initiating tablet copy from peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:19:35.811054 26931 tablet_copy_client.cc:308] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.7:38851
I20240502 08:19:35.811821 15775 tablet_copy_service.cc:140] P 9333cb10227b4801b4fa7c24e22e131b: Received BeginTabletCopySession request for tablet 815d79a2bbba49ea9e135d94c57a3390 from peer 6a0b05f3250d415590b0edf6b44fbe1f ({username='slave'} at 127.6.5.4:37117)
I20240502 08:19:35.811959 15775 tablet_copy_service.cc:161] P 9333cb10227b4801b4fa7c24e22e131b: Beginning new tablet copy session on tablet 815d79a2bbba49ea9e135d94c57a3390 from peer 6a0b05f3250d415590b0edf6b44fbe1f at {username='slave'} at 127.6.5.4:37117: session id = 6a0b05f3250d415590b0edf6b44fbe1f-815d79a2bbba49ea9e135d94c57a3390
I20240502 08:19:35.816021 15775 tablet_copy_source_session.cc:205] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:35.817260 26931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815d79a2bbba49ea9e135d94c57a3390. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:35.831449 26931 tablet_copy_client.cc:783] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:35.831764 26931 tablet_copy_client.cc:647] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:35.834355 26931 tablet_copy_client.cc:523] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:35.840145 12992 raft_consensus.cc:1203] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:35.845796 26931 tablet_bootstrap.cc:492] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:19:35.872263 26931 tablet_bootstrap.cc:492] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:35.873577 26931 tablet_bootstrap.cc:492] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap complete.
I20240502 08:19:35.873886 26931 ts_tablet_manager.cc:1367] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.004s
I20240502 08:19:35.874964 26931 raft_consensus.cc:348] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:35.875306 26931 raft_consensus.cc:729] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:19:35.876971 26931 consensus_queue.cc:260] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:35.879531 26931 ts_tablet_manager.cc:1398] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 08:19:35.880156 15775 tablet_copy_service.cc:342] P 9333cb10227b4801b4fa7c24e22e131b: Request end of tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-815d79a2bbba49ea9e135d94c57a3390 received from {username='slave'} at 127.6.5.4:37117
I20240502 08:19:35.880259 15775 tablet_copy_service.cc:433] P 9333cb10227b4801b4fa7c24e22e131b: ending tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-815d79a2bbba49ea9e135d94c57a3390 on tablet 815d79a2bbba49ea9e135d94c57a3390 with peer 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:19:36.114982 26951 ts_tablet_manager.cc:902] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3: Initiating tablet copy from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:36.158771 26951 tablet_copy_client.cc:308] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.6:44983
I20240502 08:19:36.159569 15642 tablet_copy_service.cc:140] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Received BeginTabletCopySession request for tablet 7d4a5f0c5c184b7f975f7bcc59cb6b4c from peer db7893bb29a047b3b82cc60bfd8930a3 ({username='slave'} at 127.6.5.5:56185)
I20240502 08:19:36.159715 15642 tablet_copy_service.cc:161] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Beginning new tablet copy session on tablet 7d4a5f0c5c184b7f975f7bcc59cb6b4c from peer db7893bb29a047b3b82cc60bfd8930a3 at {username='slave'} at 127.6.5.5:56185: session id = db7893bb29a047b3b82cc60bfd8930a3-7d4a5f0c5c184b7f975f7bcc59cb6b4c
I20240502 08:19:36.162712 15642 tablet_copy_source_session.cc:205] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:36.164101 26951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d4a5f0c5c184b7f975f7bcc59cb6b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:36.172081 26951 tablet_copy_client.cc:783] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:36.172289 26951 tablet_copy_client.cc:647] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:36.174211 26951 tablet_copy_client.cc:523] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:36.177582 26951 tablet_bootstrap.cc:492] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap starting.
I20240502 08:19:36.195225 26951 tablet_bootstrap.cc:492] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:36.195649 26951 tablet_bootstrap.cc:492] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3: Bootstrap complete.
I20240502 08:19:36.195923 26951 ts_tablet_manager.cc:1367] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.006s
I20240502 08:19:36.196707 26951 raft_consensus.cc:348] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:36.196996 26951 raft_consensus.cc:729] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Initialized, Role: FOLLOWER
I20240502 08:19:36.197810 26951 consensus_queue.cc:260] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:36.198570 26951 ts_tablet_manager.cc:1398] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:19:36.199429 15642 tablet_copy_service.cc:342] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Request end of tablet copy session db7893bb29a047b3b82cc60bfd8930a3-7d4a5f0c5c184b7f975f7bcc59cb6b4c received from {username='slave'} at 127.6.5.5:56185
I20240502 08:19:36.199522 15642 tablet_copy_service.cc:433] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: ending tablet copy session db7893bb29a047b3b82cc60bfd8930a3-7d4a5f0c5c184b7f975f7bcc59cb6b4c on tablet 7d4a5f0c5c184b7f975f7bcc59cb6b4c with peer db7893bb29a047b3b82cc60bfd8930a3
W20240502 08:19:36.218498 12928 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.6:44983, user_credentials={real_user=slave}} blocked reactor thread for 92037us
I20240502 08:19:36.311291 13122 raft_consensus.cc:1203] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:36.426954 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:36.450692 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:36.466255 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:36.482265 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:36.493791 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:36.515357 15487 raft_consensus.cc:1203] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:36.587996 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:36.600965 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:36.612823 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:36.730991 15619 consensus_queue.cc:237] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:36.732944 15487 raft_consensus.cc:1261] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:36.733132 12727 raft_consensus.cc:1261] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:36.733733 26395 consensus_queue.cc:1035] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:36.734112 26395 consensus_queue.cc:1035] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:36.736518 26395 raft_consensus.cc:2932] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:36.740020 15487 raft_consensus.cc:2932] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:36.744570 12583 catalog_manager.cc:5484] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index 2 to 3, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:36.753278 12727 raft_consensus.cc:2932] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:36.772183 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d4a5f0c5c184b7f975f7bcc59cb6b4c with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:51176
I20240502 08:19:36.781951 27032 tablet_replica.cc:330] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:19:36.782158 27032 raft_consensus.cc:2229] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:36.782366 27032 raft_consensus.cc:2258] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:36.782968 27032 ts_tablet_manager.cc:1875] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:36.793090 27032 ts_tablet_manager.cc:1888] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:36.793264 27032 log.cc:1196] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d4a5f0c5c184b7f975f7bcc59cb6b4c
I20240502 08:19:36.794026 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 7d4a5f0c5c184b7f975f7bcc59cb6b4c (table rebalancer_extra_table_58 [id=107c7945253947cda94a688a2c9373cd]) successfully deleted
I20240502 08:19:37.002748 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:37.014881 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:37.031388 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:37.056970 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:37.087450 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:37.120683 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:37.229735 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:37.255730 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:37.394820 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(21f25468f1b541dc9ede27359acc2b9c): perf score=0.033351
I20240502 08:19:37.464463 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(21f25468f1b541dc9ede27359acc2b9c) complete. Timing: real 0.069s user 0.049s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":334,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1454,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1581,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6865,"threads_started":2,"wal-append.queue_time_us":6723}
I20240502 08:19:37.471376 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(21f25468f1b541dc9ede27359acc2b9c): 4925 bytes on disk
I20240502 08:19:37.473309 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(21f25468f1b541dc9ede27359acc2b9c) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 08:19:37.473879 15755 consensus_queue.cc:237] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:37.482656 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(bf504f2522a147c198f99debdfe03d1e): perf score=0.033375
I20240502 08:19:37.510967 13122 raft_consensus.cc:1261] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:37.511209 15622 raft_consensus.cc:1261] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:37.511543 26900 consensus_queue.cc:1035] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:37.512094 26900 consensus_queue.cc:1035] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:37.514775 26900 raft_consensus.cc:2932] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:37.521550 12581 catalog_manager.cc:5484] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index 2 to 3, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:37.525537 13122 raft_consensus.cc:2932] T 815d79a2bbba49ea9e135d94c57a3390 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:37.543202 15622 raft_consensus.cc:2932] T 815d79a2bbba49ea9e135d94c57a3390 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:37.574772 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet 815d79a2bbba49ea9e135d94c57a3390 with delete_type TABLET_DATA_TOMBSTONED (TS e1dd14f7ca764a1490df419f13f2804a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:56258
I20240502 08:19:37.575397 27115 tablet_replica.cc:330] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:19:37.575742 27115 raft_consensus.cc:2229] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:37.576010 27115 raft_consensus.cc:2258] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:37.576671 27115 ts_tablet_manager.cc:1875] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:37.582788 27115 ts_tablet_manager.cc:1888] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:37.582948 27115 log.cc:1196] T 815d79a2bbba49ea9e135d94c57a3390 P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/815d79a2bbba49ea9e135d94c57a3390
I20240502 08:19:37.583691 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet 815d79a2bbba49ea9e135d94c57a3390 (table rebalancer_extra_table_62 [id=6aba7496ff9b4c90ae1f70fee8b918e0]) successfully deleted
I20240502 08:19:37.594151 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(21f25468f1b541dc9ede27359acc2b9c): perf score=0.033400
I20240502 08:19:37.594571 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(bf504f2522a147c198f99debdfe03d1e): perf score=0.033394
I20240502 08:19:37.629227 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(bf504f2522a147c198f99debdfe03d1e) complete. Timing: real 0.146s user 0.079s sys 0.037s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":50144,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1514,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":735,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":66880,"threads_started":2,"wal-append.queue_time_us":16917}
I20240502 08:19:37.633174 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(bf504f2522a147c198f99debdfe03d1e): 4840 bytes on disk
I20240502 08:19:37.633728 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(bf504f2522a147c198f99debdfe03d1e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 08:19:37.636911 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(809af1a2878d4cba9ab2f18c11a621a8): perf score=0.033409
I20240502 08:19:37.669831 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(bf504f2522a147c198f99debdfe03d1e) complete. Timing: real 0.075s user 0.044s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14015,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1701,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":747,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":20405,"threads_started":2,"wal-append.queue_time_us":12467}
I20240502 08:19:37.672982 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(bf504f2522a147c198f99debdfe03d1e): 4840 bytes on disk
I20240502 08:19:37.673633 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(bf504f2522a147c198f99debdfe03d1e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240502 08:19:37.707232 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(21f25468f1b541dc9ede27359acc2b9c) complete. Timing: real 0.113s user 0.056s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":427,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":2653,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":692,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":12238,"threads_started":2,"wal-append.queue_time_us":11839}
I20240502 08:19:37.709600 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(809af1a2878d4cba9ab2f18c11a621a8): perf score=0.033428
I20240502 08:19:37.757836 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(809af1a2878d4cba9ab2f18c11a621a8) complete. Timing: real 0.112s user 0.041s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1152,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":726,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7252,"threads_started":1,"wal-append.queue_time_us":7166}
I20240502 08:19:37.760701 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(809af1a2878d4cba9ab2f18c11a621a8): 4586 bytes on disk
I20240502 08:19:37.761283 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(809af1a2878d4cba9ab2f18c11a621a8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240502 08:19:37.769991 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(809af1a2878d4cba9ab2f18c11a621a8) complete. Timing: real 0.058s user 0.030s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":124,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1661,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":868,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1856,"threads_started":1,"wal-append.queue_time_us":1970}
I20240502 08:19:37.772356 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(21f25468f1b541dc9ede27359acc2b9c): 4925 bytes on disk
I20240502 08:19:37.773104 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(21f25468f1b541dc9ede27359acc2b9c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240502 08:19:37.778043 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(809af1a2878d4cba9ab2f18c11a621a8): 4586 bytes on disk
I20240502 08:19:37.778856 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(809af1a2878d4cba9ab2f18c11a621a8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240502 08:19:37.979283 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:37.983057 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:37.991070 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:38.003252 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:38.015242 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:38.035528 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:38.094731 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:38.098450 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:38.241220 13121 consensus_queue.cc:237] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:38.243726 15885 raft_consensus.cc:1261] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:38.244448 12992 raft_consensus.cc:1261] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:38.244416 26938 consensus_queue.cc:1035] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:38.245513 26938 consensus_queue.cc:1035] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:38.260885 12992 raft_consensus.cc:2932] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:38.260406 27207 raft_consensus.cc:2932] T 51fe8c8f38d4492194f6e3c83f25a60b P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:38.262423 15887 raft_consensus.cc:2932] T 51fe8c8f38d4492194f6e3c83f25a60b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:38.267459 12581 catalog_manager.cc:5484] T 51fe8c8f38d4492194f6e3c83f25a60b P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 2 to 3, VOTER db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
I20240502 08:19:38.292726 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet 51fe8c8f38d4492194f6e3c83f25a60b with delete_type TABLET_DATA_TOMBSTONED (TS db7893bb29a047b3b82cc60bfd8930a3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46880
I20240502 08:19:38.303246 27219 tablet_replica.cc:330] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3: stopping tablet replica
I20240502 08:19:38.303586 27219 raft_consensus.cc:2229] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:38.303889 27219 raft_consensus.cc:2258] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:38.304584 27219 ts_tablet_manager.cc:1875] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:38.310184 27219 ts_tablet_manager.cc:1888] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:38.310384 27219 log.cc:1196] T 51fe8c8f38d4492194f6e3c83f25a60b P db7893bb29a047b3b82cc60bfd8930a3: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-4/wal/wals/51fe8c8f38d4492194f6e3c83f25a60b
I20240502 08:19:38.311282 12573 catalog_manager.cc:4830] TS db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131): tablet 51fe8c8f38d4492194f6e3c83f25a60b (table rebalancer_dynamic_table [id=89c263bab2dc46069eb3fa03deb23175]) successfully deleted
I20240502 08:19:38.553683 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:38.565259 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:38.566689 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:38.597868 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:38.630249 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:38.697016 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:38.722491 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 08:19:38.787518 15687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.4:32979, user_credentials={real_user=slave}} blocked reactor thread for 58075.2us
I20240502 08:19:38.796274 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:38.928022 15621 consensus_queue.cc:237] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:38.930042 15887 raft_consensus.cc:1261] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:38.930193 15754 raft_consensus.cc:1261] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:38.930545 26395 consensus_queue.cc:1035] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:38.930820 26866 consensus_queue.cc:1035] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:38.933563 26395 raft_consensus.cc:2932] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:38.934316 15754 raft_consensus.cc:2932] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:38.934388 15887 raft_consensus.cc:2932] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:38.940846 12587 catalog_manager.cc:5484] T 188e9a523fee46a08a4c6180058ed923 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:38.953233 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet 188e9a523fee46a08a4c6180058ed923 with delete_type TABLET_DATA_TOMBSTONED (TS 60b6e265e7754b7187307d05ef405dd4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38740
I20240502 08:19:38.953738 27297 tablet_replica.cc:330] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:19:38.953974 27297 raft_consensus.cc:2229] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:38.954206 27297 raft_consensus.cc:2258] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:38.955175 27297 ts_tablet_manager.cc:1875] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:38.960356 27297 ts_tablet_manager.cc:1888] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:38.960539 27297 log.cc:1196] T 188e9a523fee46a08a4c6180058ed923 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/188e9a523fee46a08a4c6180058ed923
I20240502 08:19:38.961246 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet 188e9a523fee46a08a4c6180058ed923 (table rebalancer_extra_table_62 [id=6aba7496ff9b4c90ae1f70fee8b918e0]) successfully deleted
I20240502 08:19:39.269107 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:39.278764 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:39.286504 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:39.288512 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:39.306988 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:39.311161 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:39.332777 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:39.351634 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:39.532800 12730 consensus_queue.cc:237] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:39.534576 15885 raft_consensus.cc:1261] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:39.534576 15755 raft_consensus.cc:1261] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:39.535044 26965 consensus_queue.cc:1035] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:39.535307 26818 consensus_queue.cc:1035] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:39.537886 26965 raft_consensus.cc:2932] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:39.539361 15755 raft_consensus.cc:2932] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:39.543964 12584 catalog_manager.cc:5484] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 2 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 2 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:39.557711 15885 raft_consensus.cc:2932] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:39.564383 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 10ba9ba78b5444a4bc73be4f6cfd6370 with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:39.564990 27415 tablet_replica.cc:330] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:39.565275 27415 raft_consensus.cc:2229] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:39.565603 27415 raft_consensus.cc:2258] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:39.566601 27415 ts_tablet_manager.cc:1875] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:39.572629 27415 ts_tablet_manager.cc:1888] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 08:19:39.572808 27415 log.cc:1196] T 10ba9ba78b5444a4bc73be4f6cfd6370 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/10ba9ba78b5444a4bc73be4f6cfd6370
I20240502 08:19:39.573823 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 10ba9ba78b5444a4bc73be4f6cfd6370 (table rebalancer_extra_table_24 [id=efafa4d6445f46888867ab1040ffde69]) successfully deleted
I20240502 08:19:39.660518 12989 consensus_queue.cc:237] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:39.664681 13122 raft_consensus.cc:1261] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:39.664799 12861 raft_consensus.cc:1261] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:19:39.664872 12925 consensus_peers.cc:485] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 -> Peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): Couldn't send request to peer 9333cb10227b4801b4fa7c24e22e131b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea9c28b7e3b1456d96330665460a93ac. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:39.665197 26804 consensus_queue.cc:1035] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:39.665685 26804 consensus_queue.cc:1035] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:39.668984 27157 raft_consensus.cc:2932] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:39.673272 13122 raft_consensus.cc:2932] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:39.675248 12861 raft_consensus.cc:2932] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:39.677770 12585 catalog_manager.cc:5484] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 4, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:39.696612 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(0fa42a21c83942f789dcebed741b52ae): perf score=0.033362
I20240502 08:19:39.749059 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(5b29a9d8d26b4a388c89fb8e3a0fefe3): perf score=0.033378
I20240502 08:19:39.757176 15885 consensus_queue.cc:237] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:39.777467 15487 raft_consensus.cc:1261] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:39.777508 15819 consensus_peers.cc:485] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 -> Peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Couldn't send request to peer 29b035434087435cbaf4ebcac98eedee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9296dc82707f48f9ac729c2ce91529c0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:39.777844 15619 raft_consensus.cc:1261] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:39.778390 27138 consensus_queue.cc:1035] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:39.778712 26844 consensus_queue.cc:1035] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:39.781632 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(0fa42a21c83942f789dcebed741b52ae): perf score=0.033394
I20240502 08:19:39.789944 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(0fa42a21c83942f789dcebed741b52ae) complete. Timing: real 0.093s user 0.058s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3678,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1386,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":853,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":26955,"threads_started":2,"wal-append.queue_time_us":19425}
I20240502 08:19:39.797142 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(0fa42a21c83942f789dcebed741b52ae): 4925 bytes on disk
I20240502 08:19:39.797875 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(0fa42a21c83942f789dcebed741b52ae) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240502 08:19:39.801019 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(5be94bf194b04e1d965274a968f30fbf): perf score=0.033387
I20240502 08:19:39.804875 27138 raft_consensus.cc:2932] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:39.807515 15487 raft_consensus.cc:2932] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:39.813114 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(5b29a9d8d26b4a388c89fb8e3a0fefe3): perf score=0.033395
I20240502 08:19:39.813048 12588 catalog_manager.cc:5484] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:39.814386 15619 raft_consensus.cc:2932] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:39.847679 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(5b29a9d8d26b4a388c89fb8e3a0fefe3) complete. Timing: real 0.098s user 0.045s sys 0.028s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":26621,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":3953,"drs_written":1,"lbm_read_time_us":144,"lbm_reads_lt_1ms":4,"lbm_write_time_us":753,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":33718,"threads_started":2,"wal-append.queue_time_us":9218}
I20240502 08:19:39.850576 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(5b29a9d8d26b4a388c89fb8e3a0fefe3): 4840 bytes on disk
I20240502 08:19:39.851613 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(5b29a9d8d26b4a388c89fb8e3a0fefe3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240502 08:19:39.854045 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(5be94bf194b04e1d965274a968f30fbf): perf score=0.033403
W20240502 08:19:39.894028 15690 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.6:44983, user_credentials={real_user=slave}} blocked reactor thread for 61429.9us
I20240502 08:19:39.930871 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(5be94bf194b04e1d965274a968f30fbf) complete. Timing: real 0.077s user 0.062s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1264,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":750,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7027,"threads_started":1,"wal-append.queue_time_us":6929}
I20240502 08:19:39.933545 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(5be94bf194b04e1d965274a968f30fbf): 4586 bytes on disk
I20240502 08:19:39.934139 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(5be94bf194b04e1d965274a968f30fbf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 08:19:39.937445 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(5be94bf194b04e1d965274a968f30fbf) complete. Timing: real 0.136s user 0.077s sys 0.044s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1343,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8042,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":82402,"threads_started":1,"wal-append.queue_time_us":82522}
I20240502 08:19:39.940109 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(5be94bf194b04e1d965274a968f30fbf): 4586 bytes on disk
I20240502 08:19:39.940551 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(5be94bf194b04e1d965274a968f30fbf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 08:19:39.993188 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(5b29a9d8d26b4a388c89fb8e3a0fefe3) complete. Timing: real 0.180s user 0.039s sys 0.028s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":16445,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":2503,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":864,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":22752,"threads_started":2,"wal-append.queue_time_us":6483}
I20240502 08:19:39.996047 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(5b29a9d8d26b4a388c89fb8e3a0fefe3): 4840 bytes on disk
I20240502 08:19:39.996707 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(5b29a9d8d26b4a388c89fb8e3a0fefe3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 08:19:40.020644 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(0fa42a21c83942f789dcebed741b52ae) complete. Timing: real 0.239s user 0.035s sys 0.036s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":15687,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1663,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":839,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":8886,"threads_started":2,"wal-append.queue_time_us":3701}
I20240502 08:19:40.023093 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(0fa42a21c83942f789dcebed741b52ae): 4925 bytes on disk
I20240502 08:19:40.023797 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(0fa42a21c83942f789dcebed741b52ae) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240502 08:19:40.054841 15487 consensus_queue.cc:237] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:40.057605 15754 raft_consensus.cc:1261] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:40.058151 27190 consensus_queue.cc:1035] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:40.058247 12861 raft_consensus.cc:1261] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:40.061518 15420 consensus_peers.cc:485] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Couldn't send request to peer 10f0e36203914b85b77e78609ffced98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5859ab9a10054123bcf6b74bdb2f93cc. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:40.064103 27190 consensus_queue.cc:1035] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:40.076704 26875 raft_consensus.cc:2932] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:40.079913 12861 raft_consensus.cc:2932] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:40.081563 15754 raft_consensus.cc:2932] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:40.088883 12588 catalog_manager.cc:5484] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index -1 to 2, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:40.132539 15484 consensus_queue.cc:237] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:40.135921 15754 raft_consensus.cc:1261] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:40.136144 12989 raft_consensus.cc:1261] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:40.136518 27509 consensus_queue.cc:1035] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:19:40.136822 15420 consensus_peers.cc:485] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): Couldn't send request to peer 10f0e36203914b85b77e78609ffced98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df8d4e22f51b4493a0ca437352c846e7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:40.136879 26875 consensus_queue.cc:1035] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:40.143553 26875 raft_consensus.cc:2932] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:40.146844 27531 ts_tablet_manager.cc:902] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:40.147814 15754 raft_consensus.cc:2932] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:40.149677 12989 raft_consensus.cc:2932] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:40.150774 12584 catalog_manager.cc:5484] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index -1 to 2, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:40.171113 27531 tablet_copy_client.cc:308] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:19:40.171898 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet ea9c28b7e3b1456d96330665460a93ac from peer 9333cb10227b4801b4fa7c24e22e131b ({username='slave'} at 127.6.5.7:55913)
I20240502 08:19:40.172045 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet ea9c28b7e3b1456d96330665460a93ac from peer 9333cb10227b4801b4fa7c24e22e131b at {username='slave'} at 127.6.5.7:55913: session id = 9333cb10227b4801b4fa7c24e22e131b-ea9c28b7e3b1456d96330665460a93ac
I20240502 08:19:40.175711 13012 tablet_copy_source_session.cc:205] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:19:40.178369 27531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9c28b7e3b1456d96330665460a93ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:40.185958 27531 tablet_copy_client.cc:783] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 5 data blocks...
I20240502 08:19:40.201607 27531 tablet_copy_client.cc:647] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:40.205598 27531 tablet_copy_client.cc:523] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:40.210316 27531 tablet_bootstrap.cc:492] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap starting.
I20240502 08:19:40.250771 27549 ts_tablet_manager.cc:902] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee: Initiating tablet copy from peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:19:40.255287 27531 tablet_bootstrap.cc:492] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:40.255692 27531 tablet_bootstrap.cc:492] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b: Bootstrap complete.
I20240502 08:19:40.256012 27531 ts_tablet_manager.cc:1367] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b: Time spent bootstrapping tablet: real 0.046s user 0.023s sys 0.003s
I20240502 08:19:40.256773 27531 raft_consensus.cc:348] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:40.257108 27531 raft_consensus.cc:729] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333cb10227b4801b4fa7c24e22e131b, State: Initialized, Role: FOLLOWER
I20240502 08:19:40.257419 27531 consensus_queue.cc:260] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:40.258086 27531 ts_tablet_manager.cc:1398] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:19:40.259624 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session 9333cb10227b4801b4fa7c24e22e131b-ea9c28b7e3b1456d96330665460a93ac received from {username='slave'} at 127.6.5.7:55913
I20240502 08:19:40.259730 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session 9333cb10227b4801b4fa7c24e22e131b-ea9c28b7e3b1456d96330665460a93ac on tablet ea9c28b7e3b1456d96330665460a93ac with peer 9333cb10227b4801b4fa7c24e22e131b
I20240502 08:19:40.286918 27549 tablet_copy_client.cc:308] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.8:46303
I20240502 08:19:40.287565 15907 tablet_copy_service.cc:140] P 10f0e36203914b85b77e78609ffced98: Received BeginTabletCopySession request for tablet 9296dc82707f48f9ac729c2ce91529c0 from peer 29b035434087435cbaf4ebcac98eedee ({username='slave'} at 127.6.5.1:34995)
I20240502 08:19:40.287757 15907 tablet_copy_service.cc:161] P 10f0e36203914b85b77e78609ffced98: Beginning new tablet copy session on tablet 9296dc82707f48f9ac729c2ce91529c0 from peer 29b035434087435cbaf4ebcac98eedee at {username='slave'} at 127.6.5.1:34995: session id = 29b035434087435cbaf4ebcac98eedee-9296dc82707f48f9ac729c2ce91529c0
I20240502 08:19:40.289745 15907 tablet_copy_source_session.cc:205] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:40.290947 27549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9296dc82707f48f9ac729c2ce91529c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:40.297752 27549 tablet_copy_client.cc:783] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:40.297961 27549 tablet_copy_client.cc:647] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:40.300302 27549 tablet_copy_client.cc:523] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:40.306320 27549 tablet_bootstrap.cc:492] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:19:40.313556 12989 consensus_queue.cc:237] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
W20240502 08:19:40.319720 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c63cca9cd28f4d30bf901e1de1cd6461. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:40.320066 13121 raft_consensus.cc:1261] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:40.320621 27549 tablet_bootstrap.cc:492] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:40.320691 27460 consensus_queue.cc:1035] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:40.321029 27549 tablet_bootstrap.cc:492] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee: Bootstrap complete.
I20240502 08:19:40.321255 27549 ts_tablet_manager.cc:1367] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240502 08:19:40.321350 12861 raft_consensus.cc:1261] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:40.321864 27460 consensus_queue.cc:1035] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:40.321880 27549 raft_consensus.cc:348] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:40.322073 27549 raft_consensus.cc:729] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:19:40.322256 27549 consensus_queue.cc:260] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:40.322881 27549 ts_tablet_manager.cc:1398] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:19:40.329371 15907 tablet_copy_service.cc:342] P 10f0e36203914b85b77e78609ffced98: Request end of tablet copy session 29b035434087435cbaf4ebcac98eedee-9296dc82707f48f9ac729c2ce91529c0 received from {username='slave'} at 127.6.5.1:34995
I20240502 08:19:40.329501 15907 tablet_copy_service.cc:433] P 10f0e36203914b85b77e78609ffced98: ending tablet copy session 29b035434087435cbaf4ebcac98eedee-9296dc82707f48f9ac729c2ce91529c0 on tablet 9296dc82707f48f9ac729c2ce91529c0 with peer 29b035434087435cbaf4ebcac98eedee
I20240502 08:19:40.334101 15801 maintenance_manager.cc:419] P 9333cb10227b4801b4fa7c24e22e131b: Scheduling UndoDeltaBlockGCOp(ea9c28b7e3b1456d96330665460a93ac): 4586 bytes on disk
I20240502 08:19:40.335500 15691 maintenance_manager.cc:643] P 9333cb10227b4801b4fa7c24e22e131b: UndoDeltaBlockGCOp(ea9c28b7e3b1456d96330665460a93ac) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 08:19:40.338150 27460 raft_consensus.cc:2932] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:40.338793 12861 raft_consensus.cc:2932] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:40.346007 13121 raft_consensus.cc:2932] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:40.348428 12584 catalog_manager.cc:5484] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:40.407694 13121 consensus_queue.cc:237] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:40.410055 12727 raft_consensus.cc:1261] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:19:40.410400 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad53a92fbcc041378bb1958c65e997db. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:40.410692 27581 consensus_queue.cc:1035] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:40.410775 12861 raft_consensus.cc:1261] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:40.411279 27207 consensus_queue.cc:1035] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:40.414096 27207 raft_consensus.cc:2932] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:40.414844 12861 raft_consensus.cc:2932] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:40.419167 12730 raft_consensus.cc:2932] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:40.422686 12586 catalog_manager.cc:5484] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:40.473577 15755 consensus_queue.cc:237] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:40.482828 12991 raft_consensus.cc:1261] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:40.483392 15688 consensus_peers.cc:485] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b -> Peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Couldn't send request to peer 29b035434087435cbaf4ebcac98eedee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 566ef75f039f41f18b76c2e4f3e640dd. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:40.483553 15621 raft_consensus.cc:1261] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:40.483552 26901 consensus_queue.cc:1035] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:40.484447 27271 consensus_queue.cc:1035] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:40.499718 27271 raft_consensus.cc:2932] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:40.505267 12989 raft_consensus.cc:2932] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:40.510439 12584 catalog_manager.cc:5484] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:40.510732 15621 raft_consensus.cc:2932] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:40.540495 27615 ts_tablet_manager.cc:902] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:19:40.549798 12989 consensus_queue.cc:237] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:40.550750 27615 tablet_copy_client.cc:308] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:19:40.551312 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet df8d4e22f51b4493a0ca437352c846e7 from peer 10f0e36203914b85b77e78609ffced98 ({username='slave'} at 127.6.5.8:58871)
I20240502 08:19:40.551474 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet df8d4e22f51b4493a0ca437352c846e7 from peer 10f0e36203914b85b77e78609ffced98 at {username='slave'} at 127.6.5.8:58871: session id = 10f0e36203914b85b77e78609ffced98-df8d4e22f51b4493a0ca437352c846e7
I20240502 08:19:40.553097 15487 raft_consensus.cc:1261] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:40.553117 15755 raft_consensus.cc:1261] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:40.553956 27157 consensus_queue.cc:1035] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:40.554078 15508 tablet_copy_source_session.cc:205] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 08:19:40.554315 12925 consensus_peers.cc:485] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Couldn't send request to peer 6a0b05f3250d415590b0edf6b44fbe1f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fdc47843a2c42b2b1713b09acb629a7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:40.554441 27157 consensus_queue.cc:1035] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:40.555784 27615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8d4e22f51b4493a0ca437352c846e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:40.557646 27626 raft_consensus.cc:2932] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:40.558211 15755 raft_consensus.cc:2932] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:40.561915 15487 raft_consensus.cc:2932] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:40.564409 12586 catalog_manager.cc:5484] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:40.574945 27615 tablet_copy_client.cc:783] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:40.575219 27615 tablet_copy_client.cc:647] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:40.577078 27615 tablet_copy_client.cc:523] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:40.577670 15755 raft_consensus.cc:1203] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:40.580196 27615 tablet_bootstrap.cc:492] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:19:40.585566 27615 tablet_bootstrap.cc:492] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:40.585949 27615 tablet_bootstrap.cc:492] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98: Bootstrap complete.
I20240502 08:19:40.586208 27615 ts_tablet_manager.cc:1367] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:19:40.586971 27615 raft_consensus.cc:348] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:40.587244 27615 raft_consensus.cc:729] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:19:40.587493 27615 consensus_queue.cc:260] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:40.589627 27615 ts_tablet_manager.cc:1398] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 08:19:40.590602 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session 10f0e36203914b85b77e78609ffced98-df8d4e22f51b4493a0ca437352c846e7 received from {username='slave'} at 127.6.5.8:58871
I20240502 08:19:40.590739 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session 10f0e36203914b85b77e78609ffced98-df8d4e22f51b4493a0ca437352c846e7 on tablet df8d4e22f51b4493a0ca437352c846e7 with peer 10f0e36203914b85b77e78609ffced98
I20240502 08:19:40.606274 27615 ts_tablet_manager.cc:902] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:19:40.607517 27615 tablet_copy_client.cc:308] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:19:40.608089 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet 5859ab9a10054123bcf6b74bdb2f93cc from peer 10f0e36203914b85b77e78609ffced98 ({username='slave'} at 127.6.5.8:58871)
I20240502 08:19:40.608286 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet 5859ab9a10054123bcf6b74bdb2f93cc from peer 10f0e36203914b85b77e78609ffced98 at {username='slave'} at 127.6.5.8:58871: session id = 10f0e36203914b85b77e78609ffced98-5859ab9a10054123bcf6b74bdb2f93cc
I20240502 08:19:40.611160 15508 tablet_copy_source_session.cc:205] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:40.612286 27615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5859ab9a10054123bcf6b74bdb2f93cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:40.618476 27615 tablet_copy_client.cc:783] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:40.618688 27615 tablet_copy_client.cc:647] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:40.620182 27615 tablet_copy_client.cc:523] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:40.623929 27615 tablet_bootstrap.cc:492] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98: Bootstrap starting.
I20240502 08:19:40.648977 27615 tablet_bootstrap.cc:492] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:40.649377 27615 tablet_bootstrap.cc:492] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98: Bootstrap complete.
I20240502 08:19:40.649642 27615 ts_tablet_manager.cc:1367] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.007s
I20240502 08:19:40.650318 27615 raft_consensus.cc:348] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:40.650583 27615 raft_consensus.cc:729] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Initialized, Role: FOLLOWER
I20240502 08:19:40.650811 27615 consensus_queue.cc:260] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:40.655443 27615 ts_tablet_manager.cc:1398] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:19:40.657110 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session 10f0e36203914b85b77e78609ffced98-5859ab9a10054123bcf6b74bdb2f93cc received from {username='slave'} at 127.6.5.8:58871
I20240502 08:19:40.657245 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session 10f0e36203914b85b77e78609ffced98-5859ab9a10054123bcf6b74bdb2f93cc on tablet 5859ab9a10054123bcf6b74bdb2f93cc with peer 10f0e36203914b85b77e78609ffced98
I20240502 08:19:40.774981 27676 ts_tablet_manager.cc:902] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:40.778165 12730 raft_consensus.cc:1203] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:40.790633 27676 tablet_copy_client.cc:308] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:19:40.791172 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet c63cca9cd28f4d30bf901e1de1cd6461 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:58785)
I20240502 08:19:40.791296 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet c63cca9cd28f4d30bf901e1de1cd6461 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:58785: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-c63cca9cd28f4d30bf901e1de1cd6461
I20240502 08:19:40.793843 13012 tablet_copy_source_session.cc:205] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:40.796430 27676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63cca9cd28f4d30bf901e1de1cd6461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:40.814131 27676 tablet_copy_client.cc:783] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:40.814424 27676 tablet_copy_client.cc:647] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:40.819941 27676 tablet_copy_client.cc:523] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:40.823057 27676 tablet_bootstrap.cc:492] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:19:40.873178 27676 tablet_bootstrap.cc:492] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:40.873533 27676 tablet_bootstrap.cc:492] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:19:40.873766 27676 ts_tablet_manager.cc:1367] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.051s user 0.035s sys 0.002s
I20240502 08:19:40.874420 27676 raft_consensus.cc:348] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:40.874697 27676 raft_consensus.cc:729] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:19:40.874919 27676 consensus_queue.cc:260] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:40.875519 27676 ts_tablet_manager.cc:1398] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 08:19:40.876124 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-c63cca9cd28f4d30bf901e1de1cd6461 received from {username='slave'} at 127.6.5.6:58785
I20240502 08:19:40.876217 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-c63cca9cd28f4d30bf901e1de1cd6461 on tablet c63cca9cd28f4d30bf901e1de1cd6461 with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:40.902868 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:40.920928 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:40.924396 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:40.924836 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:40.949970 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:40.966094 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:40.974272 27676 ts_tablet_manager.cc:902] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979)
I20240502 08:19:41.016350 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:41.030432 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:41.042701 15885 raft_consensus.cc:1203] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:41.050652 27702 ts_tablet_manager.cc:902] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:41.063200 27676 tablet_copy_client.cc:308] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.4:32979
I20240502 08:19:41.063874 13142 tablet_copy_service.cc:140] P 6a0b05f3250d415590b0edf6b44fbe1f: Received BeginTabletCopySession request for tablet ad53a92fbcc041378bb1958c65e997db from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:37199)
I20240502 08:19:41.063994 13142 tablet_copy_service.cc:161] P 6a0b05f3250d415590b0edf6b44fbe1f: Beginning new tablet copy session on tablet ad53a92fbcc041378bb1958c65e997db from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:37199: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-ad53a92fbcc041378bb1958c65e997db
I20240502 08:19:41.066855 13142 tablet_copy_source_session.cc:205] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:41.068286 27676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad53a92fbcc041378bb1958c65e997db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:41.081769 27676 tablet_copy_client.cc:783] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:41.082129 27676 tablet_copy_client.cc:647] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:41.083515 27676 tablet_copy_client.cc:523] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:41.086732 27676 tablet_bootstrap.cc:492] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:19:41.109666 15885 raft_consensus.cc:1203] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:41.132081 27707 ts_tablet_manager.cc:902] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee: Initiating tablet copy from peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:19:41.148056 27702 tablet_copy_client.cc:308] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:19:41.148672 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet 5fdc47843a2c42b2b1713b09acb629a7 from peer 6a0b05f3250d415590b0edf6b44fbe1f ({username='slave'} at 127.6.5.4:39793)
I20240502 08:19:41.148823 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet 5fdc47843a2c42b2b1713b09acb629a7 from peer 6a0b05f3250d415590b0edf6b44fbe1f at {username='slave'} at 127.6.5.4:39793: session id = 6a0b05f3250d415590b0edf6b44fbe1f-5fdc47843a2c42b2b1713b09acb629a7
I20240502 08:19:41.151607 13012 tablet_copy_source_session.cc:205] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:41.152894 27702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fdc47843a2c42b2b1713b09acb629a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:41.160233 27702 tablet_copy_client.cc:783] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:41.160436 27702 tablet_copy_client.cc:647] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:41.161593 27707 tablet_copy_client.cc:308] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.7:38851
I20240502 08:19:41.161800 27702 tablet_copy_client.cc:523] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:41.162329 15775 tablet_copy_service.cc:140] P 9333cb10227b4801b4fa7c24e22e131b: Received BeginTabletCopySession request for tablet 566ef75f039f41f18b76c2e4f3e640dd from peer 29b035434087435cbaf4ebcac98eedee ({username='slave'} at 127.6.5.1:37709)
I20240502 08:19:41.162603 15775 tablet_copy_service.cc:161] P 9333cb10227b4801b4fa7c24e22e131b: Beginning new tablet copy session on tablet 566ef75f039f41f18b76c2e4f3e640dd from peer 29b035434087435cbaf4ebcac98eedee at {username='slave'} at 127.6.5.1:37709: session id = 29b035434087435cbaf4ebcac98eedee-566ef75f039f41f18b76c2e4f3e640dd
I20240502 08:19:41.162716 27676 tablet_bootstrap.cc:492] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:41.163136 27676 tablet_bootstrap.cc:492] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:19:41.163436 27676 ts_tablet_manager.cc:1367] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.077s user 0.040s sys 0.006s
I20240502 08:19:41.164113 27676 raft_consensus.cc:348] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:41.164330 27676 raft_consensus.cc:729] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:19:41.164567 27676 consensus_queue.cc:260] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:41.165096 27676 ts_tablet_manager.cc:1398] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:19:41.165109 27702 tablet_bootstrap.cc:492] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:19:41.165391 15775 tablet_copy_source_session.cc:205] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:41.166894 27707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566ef75f039f41f18b76c2e4f3e640dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:41.170210 13142 tablet_copy_service.cc:342] P 6a0b05f3250d415590b0edf6b44fbe1f: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-ad53a92fbcc041378bb1958c65e997db received from {username='slave'} at 127.6.5.6:37199
I20240502 08:19:41.170303 13142 tablet_copy_service.cc:433] P 6a0b05f3250d415590b0edf6b44fbe1f: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-ad53a92fbcc041378bb1958c65e997db on tablet ad53a92fbcc041378bb1958c65e997db with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:41.174335 27707 tablet_copy_client.cc:783] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:41.174612 27707 tablet_copy_client.cc:647] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:41.175930 27707 tablet_copy_client.cc:523] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:41.177647 27702 tablet_bootstrap.cc:492] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:41.178045 27702 tablet_bootstrap.cc:492] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap complete.
I20240502 08:19:41.178269 27702 ts_tablet_manager.cc:1367] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240502 08:19:41.178915 27702 raft_consensus.cc:348] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:41.179109 27702 raft_consensus.cc:729] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:19:41.179298 27702 consensus_queue.cc:260] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:41.179792 27702 ts_tablet_manager.cc:1398] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:19:41.180389 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-5fdc47843a2c42b2b1713b09acb629a7 received from {username='slave'} at 127.6.5.4:39793
I20240502 08:19:41.180490 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-5fdc47843a2c42b2b1713b09acb629a7 on tablet 5fdc47843a2c42b2b1713b09acb629a7 with peer 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:19:41.184029 27707 tablet_bootstrap.cc:492] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:19:41.188822 27707 tablet_bootstrap.cc:492] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:41.189172 27707 tablet_bootstrap.cc:492] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee: Bootstrap complete.
I20240502 08:19:41.189417 27707 ts_tablet_manager.cc:1367] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 08:19:41.190096 27707 raft_consensus.cc:348] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:41.190297 27707 raft_consensus.cc:729] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:19:41.190513 27707 consensus_queue.cc:260] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:41.192507 27707 ts_tablet_manager.cc:1398] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240502 08:19:41.193111 15775 tablet_copy_service.cc:342] P 9333cb10227b4801b4fa7c24e22e131b: Request end of tablet copy session 29b035434087435cbaf4ebcac98eedee-566ef75f039f41f18b76c2e4f3e640dd received from {username='slave'} at 127.6.5.1:37709
I20240502 08:19:41.193217 15775 tablet_copy_service.cc:433] P 9333cb10227b4801b4fa7c24e22e131b: ending tablet copy session 29b035434087435cbaf4ebcac98eedee-566ef75f039f41f18b76c2e4f3e640dd on tablet 566ef75f039f41f18b76c2e4f3e640dd with peer 29b035434087435cbaf4ebcac98eedee
I20240502 08:19:41.228601 15621 raft_consensus.cc:1203] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:41.384266 15621 raft_consensus.cc:1203] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:41.467872 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:41.510370 12727 raft_consensus.cc:1203] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:41.533149 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:41.580519 13121 raft_consensus.cc:1203] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:41.597513 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:41.597791 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:41.640774 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:41.662834 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:41.704959 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:41.750674 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:41.873658 13121 consensus_queue.cc:237] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:41.875495 15621 raft_consensus.cc:1261] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:41.875658 12727 raft_consensus.cc:1261] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:41.876009 27584 consensus_queue.cc:1035] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:41.876204 27703 consensus_queue.cc:1035] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:41.878806 27584 raft_consensus.cc:2932] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:41.879468 15621 raft_consensus.cc:2932] T ad53a92fbcc041378bb1958c65e997db P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:41.879909 12727 raft_consensus.cc:2932] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:41.886794 12579 catalog_manager.cc:5484] T ad53a92fbcc041378bb1958c65e997db P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New cstate: current_term: 1 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:41.893476 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet ad53a92fbcc041378bb1958c65e997db with delete_type TABLET_DATA_TOMBSTONED (TS e1dd14f7ca764a1490df419f13f2804a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56258
I20240502 08:19:41.894147 27806 tablet_replica.cc:330] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:19:41.894443 27806 raft_consensus.cc:2229] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:41.894857 27806 raft_consensus.cc:2258] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:41.896121 27806 ts_tablet_manager.cc:1875] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:41.905499 27806 ts_tablet_manager.cc:1888] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:41.905665 27806 log.cc:1196] T ad53a92fbcc041378bb1958c65e997db P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/ad53a92fbcc041378bb1958c65e997db
I20240502 08:19:41.906533 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet ad53a92fbcc041378bb1958c65e997db (table rebalance_test_table_19 [id=c7115c9eea2a450b944a15086ffe02bd]) successfully deleted
I20240502 08:19:42.134320 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:42.177807 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:42.185174 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:42.187335 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:42.192457 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:42.233186 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:42.235092 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:42.320752 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:42.414263 15754 consensus_queue.cc:237] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:42.416292 12992 raft_consensus.cc:1261] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:42.416484 12730 raft_consensus.cc:1261] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:42.416783 26901 consensus_queue.cc:1035] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:42.417435 26901 consensus_queue.cc:1035] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:42.419049 27271 raft_consensus.cc:2932] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:42.423122 12992 raft_consensus.cc:2932] T 566ef75f039f41f18b76c2e4f3e640dd P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:42.424863 12585 catalog_manager.cc:5484] T 566ef75f039f41f18b76c2e4f3e640dd P 9333cb10227b4801b4fa7c24e22e131b reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:42.427501 12730 raft_consensus.cc:2932] T 566ef75f039f41f18b76c2e4f3e640dd P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:42.433817 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 566ef75f039f41f18b76c2e4f3e640dd with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:42.434440 27889 tablet_replica.cc:330] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:42.434724 27889 raft_consensus.cc:2229] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:42.434962 27889 raft_consensus.cc:2258] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:42.436370 27889 ts_tablet_manager.cc:1875] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:42.442939 27889 ts_tablet_manager.cc:1888] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:42.443114 27889 log.cc:1196] T 566ef75f039f41f18b76c2e4f3e640dd P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/566ef75f039f41f18b76c2e4f3e640dd
I20240502 08:19:42.443750 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 566ef75f039f41f18b76c2e4f3e640dd (table rebalancer_extra_table_0 [id=57f8b963d2e24f929fa5234b3c552ab8]) successfully deleted
I20240502 08:19:42.695116 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:42.735357 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:42.736661 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:42.737864 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(c63cca9cd28f4d30bf901e1de1cd6461): perf score=0.033337
I20240502 08:19:42.773901 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(9366b55ca7dc48bfa5d2ed650d7a30e8): perf score=0.033338
I20240502 08:19:42.802644 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(f031f5fedb904eaf83a4902092280295): perf score=0.033351
I20240502 08:19:42.812552 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:42.813247 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:42.827347 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:42.866997 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:42.964218 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(f031f5fedb904eaf83a4902092280295): perf score=0.033397
I20240502 08:19:42.985749 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(9366b55ca7dc48bfa5d2ed650d7a30e8) complete. Timing: real 0.212s user 0.040s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":473,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1383,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":19506,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":26822,"threads_started":2,"wal-append.queue_time_us":9850}
I20240502 08:19:42.988655 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(9366b55ca7dc48bfa5d2ed650d7a30e8): 4840 bytes on disk
I20240502 08:19:42.989442 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(9366b55ca7dc48bfa5d2ed650d7a30e8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 08:19:42.992282 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:43.027539 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(f031f5fedb904eaf83a4902092280295) complete. Timing: real 0.225s user 0.070s sys 0.025s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":58976,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":2085,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":831,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":69790,"threads_started":2,"wal-append.queue_time_us":10035}
I20240502 08:19:43.030248 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(f031f5fedb904eaf83a4902092280295): 4925 bytes on disk
I20240502 08:19:43.030478 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(f031f5fedb904eaf83a4902092280295) complete. Timing: real 0.066s user 0.031s sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4228,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":8812,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":860,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11519,"threads_started":2,"wal-append.queue_time_us":7078}
I20240502 08:19:43.033464 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(f031f5fedb904eaf83a4902092280295): 4925 bytes on disk
I20240502 08:19:43.034025 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(f031f5fedb904eaf83a4902092280295) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240502 08:19:43.040107 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(c63cca9cd28f4d30bf901e1de1cd6461): perf score=0.033414
I20240502 08:19:43.047410 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(f031f5fedb904eaf83a4902092280295) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 08:19:43.049995 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(c63cca9cd28f4d30bf901e1de1cd6461): perf score=0.033414
I20240502 08:19:43.077299 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(c63cca9cd28f4d30bf901e1de1cd6461) complete. Timing: real 0.339s user 0.056s sys 0.031s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39017,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1262,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":725,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":87674,"threads_started":2,"wal-append.queue_time_us":48608}
I20240502 08:19:43.080154 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(9366b55ca7dc48bfa5d2ed650d7a30e8): perf score=0.033424
I20240502 08:19:43.140398 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(c63cca9cd28f4d30bf901e1de1cd6461) complete. Timing: real 0.082s user 0.035s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1315,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":837,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6268,"threads_started":1,"wal-append.queue_time_us":6367}
I20240502 08:19:43.142999 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(c63cca9cd28f4d30bf901e1de1cd6461): 4586 bytes on disk
I20240502 08:19:43.143587 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(c63cca9cd28f4d30bf901e1de1cd6461) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240502 08:19:43.145321 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(c63cca9cd28f4d30bf901e1de1cd6461) complete. Timing: real 0.099s user 0.034s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":1015,"drs_written":1,"lbm_read_time_us":701,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4325,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3156,"threads_started":1,"wal-append.queue_time_us":8558}
I20240502 08:19:43.148154 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(c63cca9cd28f4d30bf901e1de1cd6461): 4586 bytes on disk
I20240502 08:19:43.149031 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(c63cca9cd28f4d30bf901e1de1cd6461) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":415,"lbm_reads_lt_1ms":4}
I20240502 08:19:43.164813 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(9366b55ca7dc48bfa5d2ed650d7a30e8) complete. Timing: real 0.084s user 0.043s sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":1091,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":714,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":30500,"threads_started":1,"wal-append.queue_time_us":30585}
I20240502 08:19:43.167265 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(9366b55ca7dc48bfa5d2ed650d7a30e8): 4840 bytes on disk
I20240502 08:19:43.167812 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(9366b55ca7dc48bfa5d2ed650d7a30e8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 08:19:43.169557 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(c63cca9cd28f4d30bf901e1de1cd6461): 4586 bytes on disk
I20240502 08:19:43.170964 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(c63cca9cd28f4d30bf901e1de1cd6461) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240502 08:19:43.217227 15487 consensus_queue.cc:237] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:43.228682 15884 raft_consensus.cc:1261] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:43.228863 15755 raft_consensus.cc:1261] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:43.229382 27950 consensus_queue.cc:1035] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:43.229663 27947 consensus_queue.cc:1035] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:43.232718 27950 raft_consensus.cc:2932] T df8d4e22f51b4493a0ca437352c846e7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:43.233501 15884 raft_consensus.cc:2932] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:43.234858 15755 raft_consensus.cc:2932] T df8d4e22f51b4493a0ca437352c846e7 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:43.239702 12585 catalog_manager.cc:5484] T df8d4e22f51b4493a0ca437352c846e7 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index 2 to 3, VOTER 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3) evicted. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:43.255775 12970 tablet_service.cc:1514] Processing DeleteTablet for tablet df8d4e22f51b4493a0ca437352c846e7 with delete_type TABLET_DATA_TOMBSTONED (TS 60b6e265e7754b7187307d05ef405dd4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38740
I20240502 08:19:43.259022 27989 tablet_replica.cc:330] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4: stopping tablet replica
I20240502 08:19:43.259244 27989 raft_consensus.cc:2229] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:43.259557 27989 raft_consensus.cc:2258] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:43.260222 27989 ts_tablet_manager.cc:1875] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:43.266857 27989 ts_tablet_manager.cc:1888] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:43.266978 27989 log.cc:1196] T df8d4e22f51b4493a0ca437352c846e7 P 60b6e265e7754b7187307d05ef405dd4: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-2/wal/wals/df8d4e22f51b4493a0ca437352c846e7
I20240502 08:19:43.267583 12573 catalog_manager.cc:4830] TS 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757): tablet df8d4e22f51b4493a0ca437352c846e7 (table rebalancer_extra_table_48 [id=b1448b5e49614c108f8d76ba41e5da72]) successfully deleted
I20240502 08:19:43.582442 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:43.594708 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:43.609990 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:43.613065 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:43.640187 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:43.663509 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:43.664685 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:43.731177 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:43.843780 12992 consensus_queue.cc:237] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:43.845440 12861 raft_consensus.cc:1261] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.845486 15621 raft_consensus.cc:1261] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.845928 27949 consensus_queue.cc:1035] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.846203 27626 consensus_queue.cc:1035] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.849030 27949 raft_consensus.cc:2932] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:43.849889 15621 raft_consensus.cc:2932] T c63cca9cd28f4d30bf901e1de1cd6461 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:43.850394 12861 raft_consensus.cc:2932] T c63cca9cd28f4d30bf901e1de1cd6461 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:43.856240 12588 catalog_manager.cc:5484] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:43.875811 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet c63cca9cd28f4d30bf901e1de1cd6461 with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:19:43.884572 28087 tablet_replica.cc:330] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:19:43.884784 28087 raft_consensus.cc:2229] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:43.885072 28087 raft_consensus.cc:2258] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:43.886607 28087 ts_tablet_manager.cc:1875] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:43.894997 28087 ts_tablet_manager.cc:1888] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:43.895108 28087 log.cc:1196] T c63cca9cd28f4d30bf901e1de1cd6461 P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/c63cca9cd28f4d30bf901e1de1cd6461
I20240502 08:19:43.895817 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet c63cca9cd28f4d30bf901e1de1cd6461 (table rebalance_test_table_7 [id=7ed6dd5b8e8d45edb609b77e58525533]) successfully deleted
I20240502 08:19:44.083516 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:44.124238 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:44.133694 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:44.139773 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:44.165655 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:44.203051 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:44.263595 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:44.296533 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:44.394984 15484 consensus_queue.cc:237] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:44.396999 15887 raft_consensus.cc:1261] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:44.396989 12861 raft_consensus.cc:1261] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:44.397518 27947 consensus_queue.cc:1035] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:44.397758 28141 consensus_queue.cc:1035] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:44.400090 28145 raft_consensus.cc:2932] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:44.400705 15887 raft_consensus.cc:2932] T 5859ab9a10054123bcf6b74bdb2f93cc P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:44.400676 12861 raft_consensus.cc:2932] T 5859ab9a10054123bcf6b74bdb2f93cc P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:44.407389 12586 catalog_manager.cc:5484] T 5859ab9a10054123bcf6b74bdb2f93cc P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
I20240502 08:19:44.426039 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 5859ab9a10054123bcf6b74bdb2f93cc with delete_type TABLET_DATA_TOMBSTONED (TS 9333cb10227b4801b4fa7c24e22e131b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45134
I20240502 08:19:44.426848 28172 tablet_replica.cc:330] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:19:44.427117 28172 raft_consensus.cc:2229] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:44.427390 28172 raft_consensus.cc:2258] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:44.428112 28172 ts_tablet_manager.cc:1875] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:44.442499 28172 ts_tablet_manager.cc:1888] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:44.442649 28172 log.cc:1196] T 5859ab9a10054123bcf6b74bdb2f93cc P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/5859ab9a10054123bcf6b74bdb2f93cc
I20240502 08:19:44.443874 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 5859ab9a10054123bcf6b74bdb2f93cc (table rebalancer_extra_table_22 [id=59dbf2adb22b41a4b7b92d1b39c10d56]) successfully deleted
I20240502 08:19:44.696382 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:44.736236 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:44.745790 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:44.805608 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:44.808079 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:44.828109 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:44.858479 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:44.872066 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:45.031462 15887 consensus_queue.cc:237] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:45.033321 15487 raft_consensus.cc:1261] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:45.033720 12730 raft_consensus.cc:1261] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:45.033783 28142 consensus_queue.cc:1035] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:45.034165 27943 consensus_queue.cc:1035] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:45.035992 28142 raft_consensus.cc:2932] T 9296dc82707f48f9ac729c2ce91529c0 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:45.037067 15487 raft_consensus.cc:2932] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:45.038839 12730 raft_consensus.cc:2932] T 9296dc82707f48f9ac729c2ce91529c0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:45.043783 12586 catalog_manager.cc:5484] T 9296dc82707f48f9ac729c2ce91529c0 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:45.058436 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 9296dc82707f48f9ac729c2ce91529c0 with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:45.066727 28250 tablet_replica.cc:330] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:45.067011 28250 raft_consensus.cc:2229] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:45.067571 28250 raft_consensus.cc:2258] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:45.068157 28250 ts_tablet_manager.cc:1875] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:45.078058 28250 ts_tablet_manager.cc:1888] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:45.078200 28250 log.cc:1196] T 9296dc82707f48f9ac729c2ce91529c0 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/9296dc82707f48f9ac729c2ce91529c0
I20240502 08:19:45.078895 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 9296dc82707f48f9ac729c2ce91529c0 (table rebalancer_extra_table_36 [id=b9146919078d47a99126e22456f8f239]) successfully deleted
I20240502 08:19:45.326171 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:45.341984 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:45.375859 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:45.376706 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:45.421896 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:45.428025 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:45.485622 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:45.491634 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:45.621845 12990 consensus_queue.cc:237] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:45.627569 15754 raft_consensus.cc:1261] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:45.627655 13122 raft_consensus.cc:1261] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:45.628182 28227 consensus_queue.cc:1035] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:45.628450 28020 consensus_queue.cc:1035] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:45.630918 28227 raft_consensus.cc:2932] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:45.636493 15754 raft_consensus.cc:2932] T ea9c28b7e3b1456d96330665460a93ac P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:45.637180 12584 catalog_manager.cc:5484] T ea9c28b7e3b1456d96330665460a93ac P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:45.641404 13122 raft_consensus.cc:2932] T ea9c28b7e3b1456d96330665460a93ac P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:45.665184 12842 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9c28b7e3b1456d96330665460a93ac with delete_type TABLET_DATA_TOMBSTONED (TS e1dd14f7ca764a1490df419f13f2804a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56258
I20240502 08:19:45.665772 28350 tablet_replica.cc:330] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a: stopping tablet replica
I20240502 08:19:45.665990 28350 raft_consensus.cc:2229] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:45.666260 28350 raft_consensus.cc:2258] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:45.667421 28350 ts_tablet_manager.cc:1875] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:45.706121 28350 ts_tablet_manager.cc:1888] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:45.706439 28350 log.cc:1196] T ea9c28b7e3b1456d96330665460a93ac P e1dd14f7ca764a1490df419f13f2804a: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-1/wal/wals/ea9c28b7e3b1456d96330665460a93ac
I20240502 08:19:45.707458 12573 catalog_manager.cc:4830] TS e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): tablet ea9c28b7e3b1456d96330665460a93ac (table rebalance_test_table_1 [id=eafcd56a22d944b391206b320ed18540]) successfully deleted
I20240502 08:19:45.792711 12730 consensus_queue.cc:237] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:45.795213 15621 raft_consensus.cc:1261] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:45.795753 15887 raft_consensus.cc:1261] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:45.796739 12664 consensus_peers.cc:485] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96ab6c62a13447c38e1ec48870a822c3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:45.796825 28140 consensus_queue.cc:1035] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:45.797343 27681 consensus_queue.cc:1035] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:45.806982 27681 raft_consensus.cc:2932] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:45.807802 15621 raft_consensus.cc:2932] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:45.811973 15887 raft_consensus.cc:2932] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:45.814517 12580 catalog_manager.cc:5484] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:46.046217 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:46.069698 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:46.084610 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:46.109707 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:46.134938 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:46.135936 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:46.145972 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:46.160634 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:46.371098 28476 ts_tablet_manager.cc:902] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:46.398806 28476 tablet_copy_client.cc:308] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:19:46.399340 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet 96ab6c62a13447c38e1ec48870a822c3 from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:59845)
I20240502 08:19:46.399497 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet 96ab6c62a13447c38e1ec48870a822c3 from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:59845: session id = e1dd14f7ca764a1490df419f13f2804a-96ab6c62a13447c38e1ec48870a822c3
I20240502 08:19:46.402601 12751 tablet_copy_source_session.cc:205] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:46.405046 28476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ab6c62a13447c38e1ec48870a822c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:46.412477 28476 tablet_copy_client.cc:783] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:46.412674 28476 tablet_copy_client.cc:647] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:46.415671 28476 tablet_copy_client.cc:523] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:46.419283 28476 tablet_bootstrap.cc:492] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:19:46.424891 28476 tablet_bootstrap.cc:492] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:46.425396 28476 tablet_bootstrap.cc:492] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:19:46.425712 28476 ts_tablet_manager.cc:1367] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 08:19:46.426762 28476 raft_consensus.cc:348] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:46.427074 28476 raft_consensus.cc:729] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:19:46.427421 28476 consensus_queue.cc:260] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:46.428653 28476 ts_tablet_manager.cc:1398] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:19:46.430534 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-96ab6c62a13447c38e1ec48870a822c3 received from {username='slave'} at 127.6.5.2:59845
I20240502 08:19:46.430637 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-96ab6c62a13447c38e1ec48870a822c3 on tablet 96ab6c62a13447c38e1ec48870a822c3 with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:46.603159 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:46.603200 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:46.630079 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:46.672391 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:46.677839 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:46.718355 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:46.728971 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:46.733556 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:46.786278 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(ab808742ce8746ae8e9b8fbff435e7be): perf score=0.033353
I20240502 08:19:46.904068 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(ab808742ce8746ae8e9b8fbff435e7be) complete. Timing: real 0.117s user 0.029s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":356,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":6999,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":706,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3381,"threads_started":2,"wal-append.queue_time_us":271}
I20240502 08:19:46.906184 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(5c3f0742355442af8a29b971921b7d45): perf score=0.033379
I20240502 08:19:46.936549 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(db051ed0438742319f741a565834b644): perf score=0.033387
I20240502 08:19:46.936789 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(ab808742ce8746ae8e9b8fbff435e7be): perf score=0.033393
I20240502 08:19:46.957630 12861 raft_consensus.cc:1203] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:46.968297 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(5c3f0742355442af8a29b971921b7d45): perf score=0.033403
I20240502 08:19:46.974072 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(5c3f0742355442af8a29b971921b7d45) complete. Timing: real 0.068s user 0.050s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1337,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":793,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6375,"threads_started":1,"wal-append.queue_time_us":6267}
I20240502 08:19:46.976948 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(5c3f0742355442af8a29b971921b7d45): 4925 bytes on disk
I20240502 08:19:46.977847 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(5c3f0742355442af8a29b971921b7d45) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 08:19:46.981642 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(ab808742ce8746ae8e9b8fbff435e7be): 4840 bytes on disk
I20240502 08:19:46.982206 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(ab808742ce8746ae8e9b8fbff435e7be) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 08:19:46.997757 12992 consensus_queue.cc:237] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:47.003355 13122 raft_consensus.cc:1261] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:47.003472 15487 raft_consensus.cc:1261] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:47.003880 28539 consensus_queue.cc:1035] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:47.005990 28539 consensus_queue.cc:1035] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:47.007550 28539 raft_consensus.cc:2932] T 5fdc47843a2c42b2b1713b09acb629a7 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:47.011468 13122 raft_consensus.cc:2932] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:47.027449 12581 catalog_manager.cc:5484] T 5fdc47843a2c42b2b1713b09acb629a7 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:47.030267 15487 raft_consensus.cc:2932] T 5fdc47843a2c42b2b1713b09acb629a7 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:47.041203 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(db051ed0438742319f741a565834b644) complete. Timing: real 0.104s user 0.038s sys 0.018s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":11023,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":2777,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":816,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":18929,"threads_started":2,"wal-append.queue_time_us":7901}
I20240502 08:19:47.043896 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(db051ed0438742319f741a565834b644): 4586 bytes on disk
I20240502 08:19:47.044427 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(db051ed0438742319f741a565834b644) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240502 08:19:47.053725 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(ab808742ce8746ae8e9b8fbff435e7be) complete. Timing: real 0.114s user 0.042s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":793,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":2686,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":689,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":47082,"threads_started":2,"wal-append.queue_time_us":11604}
I20240502 08:19:47.056618 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(ab808742ce8746ae8e9b8fbff435e7be): 4840 bytes on disk
I20240502 08:19:47.059186 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(ab808742ce8746ae8e9b8fbff435e7be) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240502 08:19:47.071741 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(5c3f0742355442af8a29b971921b7d45) complete. Timing: real 0.103s user 0.053s sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":917,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":4744,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":909,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":29703,"threads_started":2,"wal-append.queue_time_us":28257}
I20240502 08:19:47.074318 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(db051ed0438742319f741a565834b644): perf score=0.033427
I20240502 08:19:47.085116 15733 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fdc47843a2c42b2b1713b09acb629a7 with delete_type TABLET_DATA_TOMBSTONED (TS 9333cb10227b4801b4fa7c24e22e131b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45134
I20240502 08:19:47.099095 28561 tablet_replica.cc:330] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b: stopping tablet replica
I20240502 08:19:47.099370 28561 raft_consensus.cc:2229] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:47.099620 28561 raft_consensus.cc:2258] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:47.100273 28561 ts_tablet_manager.cc:1875] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:47.108469 28561 ts_tablet_manager.cc:1888] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:47.108623 28561 log.cc:1196] T 5fdc47843a2c42b2b1713b09acb629a7 P 9333cb10227b4801b4fa7c24e22e131b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-6/wal/wals/5fdc47843a2c42b2b1713b09acb629a7
I20240502 08:19:47.109304 12572 catalog_manager.cc:4830] TS 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851): tablet 5fdc47843a2c42b2b1713b09acb629a7 (table rebalancer_extra_table_44 [id=6eff13bc8a8247babea96b8f0537a4cf]) successfully deleted
I20240502 08:19:47.120323 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(db051ed0438742319f741a565834b644) complete. Timing: real 0.046s user 0.039s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1218,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1030,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":496,"threads_started":1,"wal-append.queue_time_us":587}
I20240502 08:19:47.122648 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(5c3f0742355442af8a29b971921b7d45): 4925 bytes on disk
I20240502 08:19:47.125036 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(5c3f0742355442af8a29b971921b7d45) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240502 08:19:47.127485 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(db051ed0438742319f741a565834b644): 4586 bytes on disk
I20240502 08:19:47.127962 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(db051ed0438742319f741a565834b644) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 08:19:47.197243 15487 consensus_queue.cc:237] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
W20240502 08:19:47.199573 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30001d4bd6e943f88d28894ef59cbe18. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:47.199592 15884 raft_consensus.cc:1261] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:47.199993 12860 raft_consensus.cc:1261] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:47.200101 28613 consensus_queue.cc:1035] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:47.200475 28141 consensus_queue.cc:1035] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:47.205397 28613 raft_consensus.cc:2932] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:47.205901 12860 raft_consensus.cc:2932] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:47.214893 12585 catalog_manager.cc:5484] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:47.218914 15884 raft_consensus.cc:2932] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:47.259143 12990 consensus_queue.cc:237] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:47.271986 15884 raft_consensus.cc:1261] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:47.272150 15621 raft_consensus.cc:1261] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:47.272420 28551 consensus_queue.cc:1035] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:19:47.272665 12926 consensus_peers.cc:485] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Couldn't send request to peer 29b035434087435cbaf4ebcac98eedee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0b58d36dccb4112a6fcb603964e91b3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:47.272692 28539 consensus_queue.cc:1035] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:47.274890 28539 raft_consensus.cc:2932] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:47.275478 15621 raft_consensus.cc:2932] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:47.278751 15884 raft_consensus.cc:2932] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:47.280814 12579 catalog_manager.cc:5484] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:47.315899 12727 consensus_queue.cc:237] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:47.318122 13122 raft_consensus.cc:1261] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:19:47.318333 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a72d875fcb54315af393e31e743d2ae. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:47.318648 28393 consensus_queue.cc:1035] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:47.319092 12990 raft_consensus.cc:1261] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:47.319535 28393 consensus_queue.cc:1035] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:47.321771 28651 raft_consensus.cc:2932] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:47.322281 12990 raft_consensus.cc:2932] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:47.326330 13122 raft_consensus.cc:2932] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:47.328852 12581 catalog_manager.cc:5484] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index -1 to 4, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:47.372156 15621 consensus_queue.cc:237] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:47.374382 15884 raft_consensus.cc:1261] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:47.374429 13119 raft_consensus.cc:1261] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Refusing update from remote peer 8d45ab45b8eb4d03a4bb9cb88b78a98b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:47.375062 27845 consensus_queue.cc:1035] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:19:47.375341 15554 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b -> Peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933): Couldn't send request to peer e1dd14f7ca764a1490df419f13f2804a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0dede000b7943439f7b8455ea370e14. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:47.375527 28562 consensus_queue.cc:1035] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:47.377830 27845 raft_consensus.cc:2932] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:47.378654 13119 raft_consensus.cc:2932] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:47.385342 12585 catalog_manager.cc:5484] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New cstate: current_term: 1 leader_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:47.388312 15884 raft_consensus.cc:2932] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e1dd14f7ca764a1490df419f13f2804a (127.6.5.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:47.678763 28723 ts_tablet_manager.cc:902] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:19:47.695025 28724 ts_tablet_manager.cc:902] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee: Initiating tablet copy from peer 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:47.700137 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:47.702373 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:47.712842 28724 tablet_copy_client.cc:308] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.3:46757
I20240502 08:19:47.715261 13012 tablet_copy_service.cc:140] P 60b6e265e7754b7187307d05ef405dd4: Received BeginTabletCopySession request for tablet e0b58d36dccb4112a6fcb603964e91b3 from peer 29b035434087435cbaf4ebcac98eedee ({username='slave'} at 127.6.5.1:39837)
I20240502 08:19:47.715370 13012 tablet_copy_service.cc:161] P 60b6e265e7754b7187307d05ef405dd4: Beginning new tablet copy session on tablet e0b58d36dccb4112a6fcb603964e91b3 from peer 29b035434087435cbaf4ebcac98eedee at {username='slave'} at 127.6.5.1:39837: session id = 29b035434087435cbaf4ebcac98eedee-e0b58d36dccb4112a6fcb603964e91b3
I20240502 08:19:47.717339 13012 tablet_copy_source_session.cc:205] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:47.718488 28724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0b58d36dccb4112a6fcb603964e91b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:47.719687 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:47.733325 28724 tablet_copy_client.cc:783] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:47.733526 28724 tablet_copy_client.cc:647] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:47.736523 28724 tablet_copy_client.cc:523] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:47.744510 28724 tablet_bootstrap.cc:492] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
I20240502 08:19:47.761098 28723 tablet_copy_client.cc:308] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.5:44131
I20240502 08:19:47.761617 15508 tablet_copy_service.cc:140] P db7893bb29a047b3b82cc60bfd8930a3: Received BeginTabletCopySession request for tablet 30001d4bd6e943f88d28894ef59cbe18 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:47559)
I20240502 08:19:47.761744 15508 tablet_copy_service.cc:161] P db7893bb29a047b3b82cc60bfd8930a3: Beginning new tablet copy session on tablet 30001d4bd6e943f88d28894ef59cbe18 from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:47559: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-30001d4bd6e943f88d28894ef59cbe18
I20240502 08:19:47.764446 15508 tablet_copy_source_session.cc:205] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:47.765775 28723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30001d4bd6e943f88d28894ef59cbe18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:47.778903 28723 tablet_copy_client.cc:783] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:47.779225 28723 tablet_copy_client.cc:647] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:47.779386 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:47.790580 28723 tablet_copy_client.cc:523] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:47.793668 28723 tablet_bootstrap.cc:492] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:19:47.795969 28724 tablet_bootstrap.cc:492] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:47.796420 28724 tablet_bootstrap.cc:492] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee: Bootstrap complete.
I20240502 08:19:47.796767 28724 ts_tablet_manager.cc:1367] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.002s
I20240502 08:19:47.797592 28724 raft_consensus.cc:348] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:47.797885 28724 raft_consensus.cc:729] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:19:47.798149 28724 consensus_queue.cc:260] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:47.798910 28724 ts_tablet_manager.cc:1398] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:19:47.800179 13012 tablet_copy_service.cc:342] P 60b6e265e7754b7187307d05ef405dd4: Request end of tablet copy session 29b035434087435cbaf4ebcac98eedee-e0b58d36dccb4112a6fcb603964e91b3 received from {username='slave'} at 127.6.5.1:39837
I20240502 08:19:47.800280 13012 tablet_copy_service.cc:433] P 60b6e265e7754b7187307d05ef405dd4: ending tablet copy session 29b035434087435cbaf4ebcac98eedee-e0b58d36dccb4112a6fcb603964e91b3 on tablet e0b58d36dccb4112a6fcb603964e91b3 with peer 29b035434087435cbaf4ebcac98eedee
I20240502 08:19:47.805377 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:47.819255 28746 ts_tablet_manager.cc:902] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:47.836071 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:47.843609 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:47.896147 28745 ts_tablet_manager.cc:902] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a: Initiating tablet copy from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:47.897580 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:47.899875 28746 tablet_copy_client.cc:308] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.1:33183
I20240502 08:19:47.900563 12751 tablet_copy_service.cc:140] P 29b035434087435cbaf4ebcac98eedee: Received BeginTabletCopySession request for tablet 6a72d875fcb54315af393e31e743d2ae from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b ({username='slave'} at 127.6.5.6:55577)
I20240502 08:19:47.900700 12751 tablet_copy_service.cc:161] P 29b035434087435cbaf4ebcac98eedee: Beginning new tablet copy session on tablet 6a72d875fcb54315af393e31e743d2ae from peer 8d45ab45b8eb4d03a4bb9cb88b78a98b at {username='slave'} at 127.6.5.6:55577: session id = 8d45ab45b8eb4d03a4bb9cb88b78a98b-6a72d875fcb54315af393e31e743d2ae
I20240502 08:19:47.903492 12751 tablet_copy_source_session.cc:205] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:47.905571 28723 tablet_bootstrap.cc:492] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:47.905900 28723 tablet_bootstrap.cc:492] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:19:47.906190 28723 ts_tablet_manager.cc:1367] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.113s user 0.011s sys 0.000s
I20240502 08:19:47.906687 28746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a72d875fcb54315af393e31e743d2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:47.906869 28723 raft_consensus.cc:348] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:47.907095 28723 raft_consensus.cc:729] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:19:47.910140 28723 consensus_queue.cc:260] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:47.910764 28723 ts_tablet_manager.cc:1398] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:19:47.911274 15508 tablet_copy_service.cc:342] P db7893bb29a047b3b82cc60bfd8930a3: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-30001d4bd6e943f88d28894ef59cbe18 received from {username='slave'} at 127.6.5.6:47559
I20240502 08:19:47.911381 15508 tablet_copy_service.cc:433] P db7893bb29a047b3b82cc60bfd8930a3: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-30001d4bd6e943f88d28894ef59cbe18 on tablet 30001d4bd6e943f88d28894ef59cbe18 with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:47.914966 28746 tablet_copy_client.cc:783] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:47.915190 28746 tablet_copy_client.cc:647] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:47.917151 28746 tablet_copy_client.cc:523] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:47.922698 28746 tablet_bootstrap.cc:492] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap starting.
I20240502 08:19:47.958882 28745 tablet_copy_client.cc:308] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.6:44983
I20240502 08:19:47.959688 15642 tablet_copy_service.cc:140] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Received BeginTabletCopySession request for tablet d0dede000b7943439f7b8455ea370e14 from peer e1dd14f7ca764a1490df419f13f2804a ({username='slave'} at 127.6.5.2:42935)
I20240502 08:19:47.959839 15642 tablet_copy_service.cc:161] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Beginning new tablet copy session on tablet d0dede000b7943439f7b8455ea370e14 from peer e1dd14f7ca764a1490df419f13f2804a at {username='slave'} at 127.6.5.2:42935: session id = e1dd14f7ca764a1490df419f13f2804a-d0dede000b7943439f7b8455ea370e14
I20240502 08:19:47.964087 15642 tablet_copy_source_session.cc:205] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:47.965396 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0dede000b7943439f7b8455ea370e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:47.972258 28745 tablet_copy_client.cc:783] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:47.972445 28745 tablet_copy_client.cc:647] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:47.985540 28745 tablet_copy_client.cc:523] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:47.988891 28745 tablet_bootstrap.cc:492] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap starting.
I20240502 08:19:47.990396 28746 tablet_bootstrap.cc:492] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:47.990824 28746 tablet_bootstrap.cc:492] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Bootstrap complete.
I20240502 08:19:47.991075 28746 ts_tablet_manager.cc:1367] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent bootstrapping tablet: real 0.068s user 0.042s sys 0.004s
I20240502 08:19:47.991940 28746 raft_consensus.cc:348] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:47.992167 28746 raft_consensus.cc:729] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Initialized, Role: FOLLOWER
I20240502 08:19:47.992413 28746 consensus_queue.cc:260] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:47.992956 28746 ts_tablet_manager.cc:1398] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:19:47.996219 28745 tablet_bootstrap.cc:492] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:47.996263 12751 tablet_copy_service.cc:342] P 29b035434087435cbaf4ebcac98eedee: Request end of tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-6a72d875fcb54315af393e31e743d2ae received from {username='slave'} at 127.6.5.6:55577
I20240502 08:19:47.996351 12751 tablet_copy_service.cc:433] P 29b035434087435cbaf4ebcac98eedee: ending tablet copy session 8d45ab45b8eb4d03a4bb9cb88b78a98b-6a72d875fcb54315af393e31e743d2ae on tablet 6a72d875fcb54315af393e31e743d2ae with peer 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:47.996589 28745 tablet_bootstrap.cc:492] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a: Bootstrap complete.
I20240502 08:19:47.996791 28745 ts_tablet_manager.cc:1367] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 08:19:47.997588 28745 raft_consensus.cc:348] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:47.997781 28745 raft_consensus.cc:729] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Initialized, Role: FOLLOWER
I20240502 08:19:47.997998 28745 consensus_queue.cc:260] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:48.000239 28745 ts_tablet_manager.cc:1398] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 08:19:48.000878 15642 tablet_copy_service.cc:342] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Request end of tablet copy session e1dd14f7ca764a1490df419f13f2804a-d0dede000b7943439f7b8455ea370e14 received from {username='slave'} at 127.6.5.2:42935
I20240502 08:19:48.000994 15642 tablet_copy_service.cc:433] P 8d45ab45b8eb4d03a4bb9cb88b78a98b: ending tablet copy session e1dd14f7ca764a1490df419f13f2804a-d0dede000b7943439f7b8455ea370e14 on tablet d0dede000b7943439f7b8455ea370e14 with peer e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:48.058076 15619 raft_consensus.cc:1203] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:48.176019 12727 raft_consensus.cc:1203] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:48.179394 12860 raft_consensus.cc:1203] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:48.240178 15619 raft_consensus.cc:1203] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:19:48.299929 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:48.310850 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:48.336491 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:48.347190 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:48.418807 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:48.462235 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:48.516710 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 08:19:48.525282 12925 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.7:38851, user_credentials={real_user=slave}} blocked reactor thread for 101398us
I20240502 08:19:48.537361 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:48.973783 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:48.982671 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:49.010030 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:49.026772 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:49.026827 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:49.052724 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:49.057240 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 08:19:49.061748 15819 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.1:33183, user_credentials={real_user=slave}} blocked reactor thread for 97634.9us
I20240502 08:19:49.071648 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:49.237067 12730 consensus_queue.cc:237] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:49.238937 15621 raft_consensus.cc:1261] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:49.239090 12992 raft_consensus.cc:1261] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:49.239501 27681 consensus_queue.cc:1035] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:49.239748 28814 consensus_queue.cc:1035] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:49.251525 28814 raft_consensus.cc:2932] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:49.253787 12992 raft_consensus.cc:2932] T 6a72d875fcb54315af393e31e743d2ae P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:49.254820 15621 raft_consensus.cc:2932] T 6a72d875fcb54315af393e31e743d2ae P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:49.257483 12586 catalog_manager.cc:5484] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee reported cstate change: config changed from index 4 to 5, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:49.274984 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a72d875fcb54315af393e31e743d2ae with delete_type TABLET_DATA_TOMBSTONED (TS 6a0b05f3250d415590b0edf6b44fbe1f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:51176
I20240502 08:19:49.275573 28917 tablet_replica.cc:330] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f: stopping tablet replica
I20240502 08:19:49.275817 28917 raft_consensus.cc:2229] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:49.276055 28917 raft_consensus.cc:2258] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:49.276966 28917 ts_tablet_manager.cc:1875] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:49.282688 28917 ts_tablet_manager.cc:1888] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:19:49.282850 28917 log.cc:1196] T 6a72d875fcb54315af393e31e743d2ae P 6a0b05f3250d415590b0edf6b44fbe1f: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-3/wal/wals/6a72d875fcb54315af393e31e743d2ae
I20240502 08:19:49.283459 12572 catalog_manager.cc:4830] TS 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): tablet 6a72d875fcb54315af393e31e743d2ae (table rebalance_test_table_13 [id=f2c6e6573e254b7d815efdcd481a81a2]) successfully deleted
I20240502 08:19:49.571094 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:49.579223 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:49.605048 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:49.624436 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:49.625557 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:49.636605 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:49.667721 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:49.700565 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:49.778379 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(c6cd36cfde5e480cad86a046d59e8940): perf score=0.033368
I20240502 08:19:49.822727 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(a3583c839d364bae819928eca86010ec): perf score=0.033382
I20240502 08:19:49.843225 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(3c37ff40706043e9adc4da4a6da1947a): perf score=0.033397
I20240502 08:19:49.880333 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(c6cd36cfde5e480cad86a046d59e8940) complete. Timing: real 0.102s user 0.031s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":29499,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":8936,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":758,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":38594,"threads_started":2,"wal-append.queue_time_us":9056}
I20240502 08:19:49.883107 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(3c37ff40706043e9adc4da4a6da1947a): perf score=0.033392
I20240502 08:19:49.906170 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(3c37ff40706043e9adc4da4a6da1947a) complete. Timing: real 0.063s user 0.033s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":844,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":4947,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":796,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":4398,"threads_started":2,"wal-append.queue_time_us":3512}
I20240502 08:19:49.909138 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(3c37ff40706043e9adc4da4a6da1947a): 4924 bytes on disk
I20240502 08:19:49.909747 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(3c37ff40706043e9adc4da4a6da1947a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 08:19:49.911695 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(a3583c839d364bae819928eca86010ec): perf score=0.033402
I20240502 08:19:49.940775 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(a3583c839d364bae819928eca86010ec) complete. Timing: real 0.118s user 0.037s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":14786,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":6740,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":757,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":41653,"threads_started":2,"wal-append.queue_time_us":26854}
I20240502 08:19:49.944981 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(c6cd36cfde5e480cad86a046d59e8940): perf score=0.033412
I20240502 08:19:49.986493 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(a3583c839d364bae819928eca86010ec) complete. Timing: real 0.075s user 0.039s sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":17050,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":3647,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":815,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":25992,"threads_started":2,"wal-append.queue_time_us":9166}
I20240502 08:19:49.988237 12992 consensus_queue.cc:237] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:49.989434 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(a3583c839d364bae819928eca86010ec): 4586 bytes on disk
I20240502 08:19:49.989998 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(a3583c839d364bae819928eca86010ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240502 08:19:49.990588 12730 raft_consensus.cc:1261] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:49.991070 15887 raft_consensus.cc:1261] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:49.991673 28983 consensus_queue.cc:1035] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:49.992564 28983 consensus_queue.cc:1035] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:49.993755 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(3c37ff40706043e9adc4da4a6da1947a) complete. Timing: real 0.102s user 0.042s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1253,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":803,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2253,"threads_started":1,"wal-append.queue_time_us":2169}
I20240502 08:19:49.995540 28551 raft_consensus.cc:2932] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:49.996484 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(3c37ff40706043e9adc4da4a6da1947a): 4924 bytes on disk
I20240502 08:19:49.997320 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(3c37ff40706043e9adc4da4a6da1947a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 08:19:49.999140 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(c6cd36cfde5e480cad86a046d59e8940): 4840 bytes on disk
I20240502 08:19:49.999166 15887 raft_consensus.cc:2932] T e0b58d36dccb4112a6fcb603964e91b3 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:50.004339 12581 catalog_manager.cc:5484] T e0b58d36dccb4112a6fcb603964e91b3 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 1 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:50.008086 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(c6cd36cfde5e480cad86a046d59e8940) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20240502 08:19:50.023864 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet e0b58d36dccb4112a6fcb603964e91b3 with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:50.024853 29016 tablet_replica.cc:330] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:50.025135 29016 raft_consensus.cc:2229] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:50.025417 29016 raft_consensus.cc:2258] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:50.026185 29016 ts_tablet_manager.cc:1875] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:50.030725 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(c6cd36cfde5e480cad86a046d59e8940) complete. Timing: real 0.086s user 0.046s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":3032,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":850,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":247,"threads_started":1,"wal-append.queue_time_us":358}
I20240502 08:19:50.032639 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(c6cd36cfde5e480cad86a046d59e8940): 4840 bytes on disk
I20240502 08:19:50.033797 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(c6cd36cfde5e480cad86a046d59e8940) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240502 08:19:50.035751 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(a3583c839d364bae819928eca86010ec): 4586 bytes on disk
I20240502 08:19:50.036325 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(a3583c839d364bae819928eca86010ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 08:19:50.037418 12730 raft_consensus.cc:2932] T e0b58d36dccb4112a6fcb603964e91b3 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:50.039829 29016 ts_tablet_manager.cc:1888] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:50.039942 29016 log.cc:1196] T e0b58d36dccb4112a6fcb603964e91b3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/e0b58d36dccb4112a6fcb603964e91b3
I20240502 08:19:50.040568 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet e0b58d36dccb4112a6fcb603964e91b3 (table rebalancer_extra_table_50 [id=ac4fff256cff4b5fbc1f66de9625967c]) successfully deleted
I20240502 08:19:50.305683 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:50.313069 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:50.318325 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:50.319473 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:50.333428 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:50.353516 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:50.410683 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:50.470080 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:50.588269 15487 consensus_queue.cc:237] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:50.590211 15621 raft_consensus.cc:1261] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:50.590319 12862 raft_consensus.cc:1261] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:50.590873 29028 consensus_queue.cc:1035] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:50.591150 28613 consensus_queue.cc:1035] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:50.593420 28613 raft_consensus.cc:2932] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:50.594002 15621 raft_consensus.cc:2932] T 30001d4bd6e943f88d28894ef59cbe18 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:50.594576 12862 raft_consensus.cc:2932] T 30001d4bd6e943f88d28894ef59cbe18 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:50.600129 12580 catalog_manager.cc:5484] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: config changed from index 2 to 3, VOTER 10f0e36203914b85b77e78609ffced98 (127.6.5.8) evicted. New cstate: current_term: 1 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:50.616885 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet 30001d4bd6e943f88d28894ef59cbe18 with delete_type TABLET_DATA_TOMBSTONED (TS 10f0e36203914b85b77e78609ffced98 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:47272
I20240502 08:19:50.617445 29113 tablet_replica.cc:330] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98: stopping tablet replica
I20240502 08:19:50.617631 29113 raft_consensus.cc:2229] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:50.617803 29113 raft_consensus.cc:2258] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:50.618379 29113 ts_tablet_manager.cc:1875] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:50.623186 29113 ts_tablet_manager.cc:1888] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:50.623489 29113 log.cc:1196] T 30001d4bd6e943f88d28894ef59cbe18 P 10f0e36203914b85b77e78609ffced98: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-7/wal/wals/30001d4bd6e943f88d28894ef59cbe18
I20240502 08:19:50.624934 12574 catalog_manager.cc:4830] TS 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303): tablet 30001d4bd6e943f88d28894ef59cbe18 (table rebalancer_extra_table_60 [id=94d0e1dcc3fb43e694fe94acec28e365]) successfully deleted
I20240502 08:19:50.879886 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:50.887322 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:50.921608 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:50.930985 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:50.936693 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:50.942940 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:50.966866 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:51.022953 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:51.166767 12730 consensus_queue.cc:237] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:51.168772 12862 raft_consensus.cc:1261] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:51.168989 15887 raft_consensus.cc:1261] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:51.169226 28907 consensus_queue.cc:1035] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:51.169497 29035 consensus_queue.cc:1035] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:51.171849 28907 raft_consensus.cc:2932] T 96ab6c62a13447c38e1ec48870a822c3 P 29b035434087435cbaf4ebcac98eedee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:51.172489 12862 raft_consensus.cc:2932] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:51.185941 12585 catalog_manager.cc:5484] T 96ab6c62a13447c38e1ec48870a822c3 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New cstate: current_term: 1 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:51.191748 15887 raft_consensus.cc:2932] T 96ab6c62a13447c38e1ec48870a822c3 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:51.205083 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ab6c62a13447c38e1ec48870a822c3 with delete_type TABLET_DATA_TOMBSTONED (TS 8d45ab45b8eb4d03a4bb9cb88b78a98b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43810
I20240502 08:19:51.217706 29205 tablet_replica.cc:330] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: stopping tablet replica
I20240502 08:19:51.218014 29205 raft_consensus.cc:2229] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:51.218284 29205 raft_consensus.cc:2258] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:51.219008 29205 ts_tablet_manager.cc:1875] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:51.224045 29205 ts_tablet_manager.cc:1888] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240502 08:19:51.224139 29205 log.cc:1196] T 96ab6c62a13447c38e1ec48870a822c3 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Deleting WAL directory at /tmp/dist-test-taskTc4cwt/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714637794771537-6164-0/raft_consensus-itest-cluster/ts-5/wal/wals/96ab6c62a13447c38e1ec48870a822c3
I20240502 08:19:51.225114 12575 catalog_manager.cc:4830] TS 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): tablet 96ab6c62a13447c38e1ec48870a822c3 (table rebalancer_extra_table_16 [id=ac73d0460aa5473a8d5bdc70b924fe3f]) successfully deleted
I20240502 08:19:51.570713 15600 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:51.576102 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:51.579756 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:51.595499 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:51.597121 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:51.644019 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:51.735985 12842 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 08:19:51.739229 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:51.844910 15621 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0dede000b7943439f7b8455ea370e14"
dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54922
I20240502 08:19:51.845072 15621 raft_consensus.cc:595] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:51.870119 28562 raft_consensus.cc:979] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: : Instructing follower e1dd14f7ca764a1490df419f13f2804a to start an election
I20240502 08:19:51.870239 28562 raft_consensus.cc:1067] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Signalling peer e1dd14f7ca764a1490df419f13f2804a to start an election
I20240502 08:19:51.870811 12861 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0dede000b7943439f7b8455ea370e14"
dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
from {username='slave'} at 127.6.5.6:60525
I20240502 08:19:51.870992 12861 raft_consensus.cc:482] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:51.871078 12861 raft_consensus.cc:3037] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:51.873564 12861 raft_consensus.cc:504] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:51.874828 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0dede000b7943439f7b8455ea370e14" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:19:51.874943 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0dede000b7943439f7b8455ea370e14" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:19:51.875018 15887 raft_consensus.cc:3037] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:51.875126 13122 raft_consensus.cc:3037] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:51.875065 15621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0dede000b7943439f7b8455ea370e14" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b"
I20240502 08:19:51.875226 15621 raft_consensus.cc:3032] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:51.875273 15621 raft_consensus.cc:729] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d45ab45b8eb4d03a4bb9cb88b78a98b, State: Running, Role: LEADER
I20240502 08:19:51.875473 15621 consensus_queue.cc:260] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:51.875792 15621 raft_consensus.cc:3037] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:51.877481 15887 raft_consensus.cc:2454] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 2.
I20240502 08:19:51.877630 13122 raft_consensus.cc:2454] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 2.
I20240502 08:19:51.878090 15621 raft_consensus.cc:2454] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 2.
I20240502 08:19:51.878647 12798 leader_election.cc:302] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:19:51.879221 28676 raft_consensus.cc:2781] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:51.879355 28676 raft_consensus.cc:686] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [term 2 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:19:51.879676 12861 leader_election.cc:288] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:51.879601 28676 consensus_queue.cc:237] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:51.884903 12579 catalog_manager.cc:5484] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 2 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:52.012035 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(b87724cd8beb4449b272f972c85a6666): perf score=0.033354
I20240502 08:19:52.028568 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(6a72d875fcb54315af393e31e743d2ae): perf score=0.033355
I20240502 08:19:52.054323 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(91f185fbf27049a7992daa1d905e5575): perf score=0.033365
I20240502 08:19:52.107918 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(b87724cd8beb4449b272f972c85a6666) complete. Timing: real 0.096s user 0.047s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":29240,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":3978,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":922,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":34710,"threads_started":2,"wal-append.queue_time_us":7281}
I20240502 08:19:52.111711 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(b87724cd8beb4449b272f972c85a6666): 4586 bytes on disk
I20240502 08:19:52.122258 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(b87724cd8beb4449b272f972c85a6666) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 08:19:52.124938 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(6a72d875fcb54315af393e31e743d2ae): perf score=0.033382
I20240502 08:19:52.152180 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:52.162034 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:52.190635 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(91f185fbf27049a7992daa1d905e5575): perf score=0.033402
I20240502 08:19:52.204883 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:52.217700 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:52.235916 15600 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:52.251626 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(91f185fbf27049a7992daa1d905e5575) complete. Timing: real 0.197s user 0.056s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":12670,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1670,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":781,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":86887,"threads_started":2,"wal-append.queue_time_us":55520}
I20240502 08:19:52.254161 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(91f185fbf27049a7992daa1d905e5575): 4840 bytes on disk
I20240502 08:19:52.256304 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(91f185fbf27049a7992daa1d905e5575) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240502 08:19:52.258824 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(6a72d875fcb54315af393e31e743d2ae) complete. Timing: real 0.230s user 0.047s sys 0.022s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":21002,"dirs.run_cpu_time_us":135,"dirs.run_wall_time_us":986,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":917,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":48986,"threads_started":2,"wal-append.queue_time_us":28213}
I20240502 08:19:52.266145 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(6a72d875fcb54315af393e31e743d2ae): 4925 bytes on disk
I20240502 08:19:52.267179 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(6a72d875fcb54315af393e31e743d2ae) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 08:19:52.270076 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(b87724cd8beb4449b272f972c85a6666): perf score=0.033420
I20240502 08:19:52.284834 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:52.329519 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:52.329654 12842 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:19:52.340349 13122 raft_consensus.cc:1261] T d0dede000b7943439f7b8455ea370e14 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:19:52.341085 28676 consensus_queue.cc:1035] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:52.347760 15884 raft_consensus.cc:1261] T d0dede000b7943439f7b8455ea370e14 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:19:52.348295 28676 consensus_queue.cc:1035] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:52.372109 15619 raft_consensus.cc:1261] T d0dede000b7943439f7b8455ea370e14 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 2 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 08:19:52.372648 29170 consensus_queue.cc:1035] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:52.436259 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(91f185fbf27049a7992daa1d905e5575) complete. Timing: real 0.245s user 0.033s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5012,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":4236,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":822,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9892,"threads_started":2,"wal-append.queue_time_us":326}
I20240502 08:19:52.438932 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(91f185fbf27049a7992daa1d905e5575): 4840 bytes on disk
I20240502 08:19:52.439579 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(91f185fbf27049a7992daa1d905e5575) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 08:19:52.443328 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(b87724cd8beb4449b272f972c85a6666) complete. Timing: real 0.170s user 0.034s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1147,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2881,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":190,"threads_started":1,"wal-append.queue_time_us":286}
I20240502 08:19:52.445461 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(b87724cd8beb4449b272f972c85a6666): 4586 bytes on disk
I20240502 08:19:52.446220 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(b87724cd8beb4449b272f972c85a6666) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240502 08:19:52.503661 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(6a72d875fcb54315af393e31e743d2ae) complete. Timing: real 0.379s user 0.039s sys 0.025s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":87,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1389,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":868,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13391,"threads_started":1,"wal-append.queue_time_us":7389}
I20240502 08:19:52.506198 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(6a72d875fcb54315af393e31e743d2ae): 4925 bytes on disk
I20240502 08:19:52.506752 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(6a72d875fcb54315af393e31e743d2ae) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240502 08:19:52.579742 12862 consensus_queue.cc:237] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:52.581751 15619 raft_consensus.cc:1261] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:52.581967 12795 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Couldn't send request to peer 6a0b05f3250d415590b0edf6b44fbe1f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 495ea7de81d34b50a92f6cb52795fadf. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:52.582147 15487 raft_consensus.cc:1261] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:52.582671 29170 consensus_queue.cc:1035] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:52.582955 28676 consensus_queue.cc:1035] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:52.585347 29170 raft_consensus.cc:2932] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:52.586182 15619 raft_consensus.cc:2932] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:52.592829 15487 raft_consensus.cc:2932] T 495ea7de81d34b50a92f6cb52795fadf P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:52.593677 12580 catalog_manager.cc:5484] T 495ea7de81d34b50a92f6cb52795fadf P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:52.624691 15884 consensus_queue.cc:237] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
W20240502 08:19:52.627471 15819 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183): Couldn't send request to peer 29b035434087435cbaf4ebcac98eedee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01b4100f5ab3405cabd93cff75f61822. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:52.627475 15487 raft_consensus.cc:1261] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:52.628154 15619 raft_consensus.cc:1261] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:52.628731 29412 consensus_queue.cc:1035] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:52.629187 28884 consensus_queue.cc:1035] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:52.635396 29412 raft_consensus.cc:2932] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:52.636340 15621 raft_consensus.cc:2932] T 01b4100f5ab3405cabd93cff75f61822 P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:52.640827 15487 raft_consensus.cc:2932] T 01b4100f5ab3405cabd93cff75f61822 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:52.642302 12586 catalog_manager.cc:5484] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index -1 to 2, VOTER 29b035434087435cbaf4ebcac98eedee (127.6.5.1) added. New cstate: current_term: 1 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:52.701174 12860 consensus_queue.cc:237] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:52.703744 15884 raft_consensus.cc:1261] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:52.704500 29170 consensus_queue.cc:1035] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:19:52.705480 12798 consensus_peers.cc:485] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c28e9d3d29548a894b520ae7b324129. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:52.707247 15754 raft_consensus.cc:1261] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:52.707903 28676 consensus_queue.cc:1035] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:52.713372 29170 raft_consensus.cc:2932] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:52.717126 15754 raft_consensus.cc:2932] T 8c28e9d3d29548a894b520ae7b324129 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:52.717969 15884 raft_consensus.cc:2932] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:52.725313 12587 catalog_manager.cc:5484] T 8c28e9d3d29548a894b520ae7b324129 P 10f0e36203914b85b77e78609ffced98 reported cstate change: config changed from index -1 to 2, VOTER 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) added. New cstate: current_term: 1 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:52.759231 15754 consensus_queue.cc:237] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:52.763974 15621 raft_consensus.cc:1261] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:52.764467 15487 raft_consensus.cc:1261] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Refusing update from remote peer 9333cb10227b4801b4fa7c24e22e131b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 08:19:52.765059 15687 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979): Couldn't send request to peer 6a0b05f3250d415590b0edf6b44fbe1f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aadd5844b106485e95ecf6fa923cd6ea. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:52.765312 28449 consensus_queue.cc:1035] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:52.766533 29450 consensus_queue.cc:1035] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:52.769791 28449 raft_consensus.cc:2932] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:52.771457 15621 raft_consensus.cc:2932] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:52.774937 15484 raft_consensus.cc:2932] T aadd5844b106485e95ecf6fa923cd6ea P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:52.779237 12580 catalog_manager.cc:5484] T aadd5844b106485e95ecf6fa923cd6ea P 8d45ab45b8eb4d03a4bb9cb88b78a98b reported cstate change: config changed from index -1 to 2, VOTER 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4) added. New cstate: current_term: 1 leader_uuid: "9333cb10227b4801b4fa7c24e22e131b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } }
I20240502 08:19:53.012867 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:53.021860 29495 ts_tablet_manager.cc:902] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f: Initiating tablet copy from peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:19:53.040155 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:53.061002 29495 tablet_copy_client.cc:308] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.2:36933
I20240502 08:19:53.061599 12882 tablet_copy_service.cc:140] P e1dd14f7ca764a1490df419f13f2804a: Received BeginTabletCopySession request for tablet 495ea7de81d34b50a92f6cb52795fadf from peer 6a0b05f3250d415590b0edf6b44fbe1f ({username='slave'} at 127.6.5.4:60905)
I20240502 08:19:53.061743 12882 tablet_copy_service.cc:161] P e1dd14f7ca764a1490df419f13f2804a: Beginning new tablet copy session on tablet 495ea7de81d34b50a92f6cb52795fadf from peer 6a0b05f3250d415590b0edf6b44fbe1f at {username='slave'} at 127.6.5.4:60905: session id = 6a0b05f3250d415590b0edf6b44fbe1f-495ea7de81d34b50a92f6cb52795fadf
I20240502 08:19:53.064471 12882 tablet_copy_source_session.cc:205] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:53.065786 29495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 495ea7de81d34b50a92f6cb52795fadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:53.070219 15600 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:53.075047 29495 tablet_copy_client.cc:783] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:53.075258 29495 tablet_copy_client.cc:647] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:53.075260 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:53.091459 29495 tablet_copy_client.cc:523] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:53.106418 29495 tablet_bootstrap.cc:492] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
I20240502 08:19:53.116827 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:53.123917 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:53.136905 12842 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:19:53.163014 29495 tablet_bootstrap.cc:492] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:53.163374 29495 tablet_bootstrap.cc:492] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap complete.
I20240502 08:19:53.163612 29495 ts_tablet_manager.cc:1367] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.004s
I20240502 08:19:53.164278 29495 raft_consensus.cc:348] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:53.164505 29495 raft_consensus.cc:729] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:19:53.177359 29516 ts_tablet_manager.cc:902] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f: Initiating tablet copy from peer 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
I20240502 08:19:53.178695 29516 tablet_copy_client.cc:308] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.7:38851
I20240502 08:19:53.179294 15775 tablet_copy_service.cc:140] P 9333cb10227b4801b4fa7c24e22e131b: Received BeginTabletCopySession request for tablet aadd5844b106485e95ecf6fa923cd6ea from peer 6a0b05f3250d415590b0edf6b44fbe1f ({username='slave'} at 127.6.5.4:37117)
I20240502 08:19:53.179428 15775 tablet_copy_service.cc:161] P 9333cb10227b4801b4fa7c24e22e131b: Beginning new tablet copy session on tablet aadd5844b106485e95ecf6fa923cd6ea from peer 6a0b05f3250d415590b0edf6b44fbe1f at {username='slave'} at 127.6.5.4:37117: session id = 6a0b05f3250d415590b0edf6b44fbe1f-aadd5844b106485e95ecf6fa923cd6ea
I20240502 08:19:53.183375 29514 ts_tablet_manager.cc:902] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee: Initiating tablet copy from peer 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:19:53.183658 29515 ts_tablet_manager.cc:902] T 8c28e9d3d29548a894b520ae7b324129 P 8d45ab45b8eb4d03a4bb9cb88b78a98b: Initiating tablet copy from peer e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
F20240502 08:19:53.183944 29515 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-30001d4bd6e943f88d28894ef59cbe18.queue_time_us, tablet-download-pool-30001d4bd6e943f88d28894ef59cbe18.queue_time_us) (tablet-download-pool-30001d4bd6e943f88d28894ef59cbe18.run_cpu_time_us, tablet-download-pool-30001d4bd6e943f88d28894ef59cbe18.run_cpu_time_us) (tablet-download-pool-30001d4bd6e943f88d28894ef59cbe18.run_wall_time_us, tablet-download-pool-30001d4bd6e943f88d28894ef59cbe18.run_wall_time_us) (tablet-download-pool-3c37ff40706043e9adc4da4a6da1947a.queue_time_us, tablet-download-pool-3c37ff40706043e9adc4da4a6da1947a.queue_time_us) (tablet-download-pool-3c37ff40706043e9adc4da4a6da1947a.run_cpu_time_us, tablet-download-pool-3c37ff40706043e9adc4da4a6da1947a.run_cpu_time_us) (tablet-download-pool-3c37ff40706043e9adc4da4a6da1947a.run_wall_time_us, tablet-download-pool-3c37ff40706043e9adc4da4a6da1947a.run_wall_time_us) (tablet-download-pool-553cec4eb28341dba84afc79b540ae4b.queue_time_us, tablet-download-pool-553cec4eb28341dba84afc79b540ae4b.queue_time_us) (tablet-download-pool-553cec4eb28341dba84afc79b540ae4b.run_cpu_time_us, tablet-download-pool-553cec4eb28341dba84afc79b540ae4b.run_cpu_time_us) (tablet-download-pool-553cec4eb28341dba84afc79b540ae4b.run_wall_time_us, tablet-download-pool-553cec4eb28341dba84afc79b540ae4b.run_wall_time_us) (tablet-download-pool-5b29a9d8d26b4a388c89fb8e3a0fefe3.queue_time_us, tablet-download-pool-5b29a9d8d26b4a388c89fb8e3a0fefe3.queue_time_us) (tablet-download-pool-5b29a9d8d26b4a388c89fb8e3a0fefe3.run_cpu_time_us, tablet-download-pool-5b29a9d8d26b4a388c89fb8e3a0fefe3.run_cpu_time_us) (tablet-download-pool-5b29a9d8d26b4a388c89fb8e3a0fefe3.run_wall_time_us, tablet-download-pool-5b29a9d8d26b4a388c89fb8e3a0fefe3.run_wall_time_us) (tablet-download-pool-6a72d875fcb54315af393e31e743d2ae.queue_time_us, tablet-download-pool-6a72d875fcb54315af393e31e743d2ae.queue_time_us) (tablet-download-pool-6a72d875fcb54315af393e31e743d2ae.run_cpu_time_us, tablet-download-pool-6a72d875fcb54315af393e31e743d2ae.run_cpu_time_us) (tablet-download-pool-6a72d875fcb54315af393e31e743d2ae.run_wall_time_us, tablet-download-pool-6a72d875fcb54315af393e31e743d2ae.run_wall_time_us) (tablet-download-pool-7a8e5440619a4cb0b40dd286862184cd.queue_time_us, tablet-download-pool-7a8e5440619a4cb0b40dd286862184cd.queue_time_us) (tablet-download-pool-7a8e5440619a4cb0b40dd286862184cd.run_cpu_time_us, tablet-download-pool-7a8e5440619a4cb0b40dd286862184cd.run_cpu_time_us) (tablet-download-pool-7a8e5440619a4cb0b40dd286862184cd.run_wall_time_us, tablet-download-pool-7a8e5440619a4cb0b40dd286862184cd.run_wall_time_us) (tablet-download-pool-809af1a2878d4cba9ab2f18c11a621a8.queue_time_us, tablet-download-pool-809af1a2878d4cba9ab2f18c11a621a8.queue_time_us) (tablet-download-pool-809af1a2878d4cba9ab2f18c11a621a8.run_cpu_time_us, tablet-download-pool-809af1a2878d4cba9ab2f18c11a621a8.run_cpu_time_us) (tablet-download-pool-809af1a2878d4cba9ab2f18c11a621a8.run_wall_time_us, tablet-download-pool-809af1a2878d4cba9ab2f18c11a621a8.run_wall_time_us) (tablet-download-pool-83e0c78cb2374ba483315ca8a827e532.queue_time_us, tablet-download-pool-83e0c78cb2374ba483315ca8a827e532.queue_time_us) (tablet-download-pool-83e0c78cb2374ba483315ca8a827e532.run_cpu_time_us, tablet-download-pool-83e0c78cb2374ba483315ca8a827e532.run_cpu_time_us) (tablet-download-pool-83e0c78cb2374ba483315ca8a827e532.run_wall_time_us, tablet-download-pool-83e0c78cb2374ba483315ca8a827e532.run_wall_time_us) (tablet-download-pool-8c28e9d3d29548a894b520ae7b324129.queue_time_us, tablet-download-pool-8c28e9d3d29548a894b520ae7b324129.queue_time_us) (tablet-download-pool-9a55a182e1f54db698ba8060eea80da0.queue_time_us, tablet-download-pool-9a55a182e1f54db698ba8060eea80da0.queue_time_us) (tablet-download-pool-9a55a182e1f54db698ba8060eea80da0.run_cpu_time_us, tablet-download-pool-9a55a182e1f54db698ba8060eea80da0.run_cpu_time_us) (tablet-download-pool-9a55a182e1f54db698ba8060eea80da0.run_wall_time_us, tablet-download-pool-9a55a182e1f54db698ba8060eea80da0.run_wall_time_us) (tablet-download-pool-ab808742ce8746ae8e9b8fbff435e7be.queue_time_us, tablet-download-pool-ab808742ce8746ae8e9b8fbff435e7be.queue_time_us) (tablet-download-pool-ab808742ce8746ae8e9b8fbff435e7be.run_cpu_time_us, tablet-download-pool-ab808742ce8746ae8e9b8fbff435e7be.run_cpu_time_us) (tablet-download-pool-ab808742ce8746ae8e9b8fbff435e7be.run_wall_time_us, tablet-download-pool-ab808742ce8746ae8e9b8fbff435e7be.run_wall_time_us) (tablet-download-pool-ad53a92fbcc041378bb1958c65e997db.queue_time_us, tablet-download-pool-ad53a92fbcc041378bb1958c65e997db.queue_time_us) (tablet-download-pool-ad53a92fbcc041378bb1958c65e997db.run_cpu_time_us, tablet-download-pool-ad53a92fbcc041378bb1958c65e997db.run_cpu_time_us) (tablet-download-pool-ad53a92fbcc041378bb1958c65e997db.run_wall_time_us, tablet-download-pool-ad53a92fbcc041378bb1958c65e997db.run_wall_time_us) (tablet-download-pool-c60b24cf2d4740ae9b8829e288bc2530.queue_time_us, tablet-download-pool-c60b24cf2d4740ae9b8829e288bc2530.queue_time_us) (tablet-download-pool-c60b24cf2d4740ae9b8829e288bc2530.run_cpu_time_us, tablet-download-pool-c60b24cf2d4740ae9b8829e288bc2530.run_cpu_time_us) (tablet-download-pool-c60b24cf2d4740ae9b8829e288bc2530.run_wall_time_us, tablet-download-pool-c60b24cf2d4740ae9b8829e288bc2530.run_wall_time_us) (tablet-download-pool-c63cca9cd28f4d30bf901e1de1cd6461.queue_time_us, tablet-download-pool-c63cca9cd28f4d30bf901e1de1cd6461.queue_time_us) (tablet-download-pool-c63cca9cd28f4d30bf901e1de1cd6461.run_cpu_time_us, tablet-download-pool-c63cca9cd28f4d30bf901e1de1cd6461.run_cpu_time_us) (tablet-download-pool-c63cca9cd28f4d30bf901e1de1cd6461.run_wall_time_us, tablet-download-pool-c63cca9cd28f4d30bf901e1de1cd6461.run_wall_time_us) (tablet-download-pool-c9ad7bf820c84aae9a8875fb92f8e5d5.queue_time_us, tablet-download-pool-c9ad7bf820c84aae9a8875fb92f8e5d5.queue_time_us) (tablet-download-pool-c9ad7bf820c84aae9a8875fb92f8e5d5.run_cpu_time_us, tablet-download-pool-c9ad7bf820c84aae9a8875fb92f8e5d5.run_cpu_time_us) (tablet-download-pool-c9ad7bf820c84aae9a8875fb92f8e5d5.run_wall_time_us, tablet-download-pool-c9ad7bf820c84aae9a8875fb92f8e5d5.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 08:19:53.185046 15775 tablet_copy_source_session.cc:205] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:53.188324 29516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadd5844b106485e95ecf6fa923cd6ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:53.189486 29495 consensus_queue.cc:260] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:53.190034 29495 ts_tablet_manager.cc:1398] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.026s user 0.002s sys 0.001s
I20240502 08:19:53.192109 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:53.192970 12882 tablet_copy_service.cc:342] P e1dd14f7ca764a1490df419f13f2804a: Request end of tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-495ea7de81d34b50a92f6cb52795fadf received from {username='slave'} at 127.6.5.4:60905
I20240502 08:19:53.193140 12882 tablet_copy_service.cc:433] P e1dd14f7ca764a1490df419f13f2804a: ending tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-495ea7de81d34b50a92f6cb52795fadf on tablet 495ea7de81d34b50a92f6cb52795fadf with peer 6a0b05f3250d415590b0edf6b44fbe1f
@ 0x7f84a9502dcd google::LogMessage::Fail() at ??:0
@ 0x7f84a9506b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f84a95027cc google::LogMessage::Flush() at ??:0
@ 0x7f84a9503f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f84ac243742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f84ac20ff8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 08:19:53.242358 29516 tablet_copy_client.cc:783] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:53.242601 29516 tablet_copy_client.cc:647] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:53.244025 29516 tablet_copy_client.cc:523] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f84ac209cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f84bb65c8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f84bb674daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 08:19:53.265645 29516 tablet_bootstrap.cc:492] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap starting.
@ 0x7f84bb79f480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 08:19:53.277282 29514 tablet_copy_client.cc:308] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Beginning tablet copy session from remote peer at address 127.6.5.8:46303
I20240502 08:19:53.277797 15907 tablet_copy_service.cc:140] P 10f0e36203914b85b77e78609ffced98: Received BeginTabletCopySession request for tablet 01b4100f5ab3405cabd93cff75f61822 from peer 29b035434087435cbaf4ebcac98eedee ({username='slave'} at 127.6.5.1:34995)
I20240502 08:19:53.277927 15907 tablet_copy_service.cc:161] P 10f0e36203914b85b77e78609ffced98: Beginning new tablet copy session on tablet 01b4100f5ab3405cabd93cff75f61822 from peer 29b035434087435cbaf4ebcac98eedee at {username='slave'} at 127.6.5.1:34995: session id = 29b035434087435cbaf4ebcac98eedee-01b4100f5ab3405cabd93cff75f61822
I20240502 08:19:53.280607 15907 tablet_copy_source_session.cc:205] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:53.281692 29514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b4100f5ab3405cabd93cff75f61822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:53.283712 29516 tablet_bootstrap.cc:492] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:53.284004 29516 tablet_bootstrap.cc:492] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f: Bootstrap complete.
I20240502 08:19:53.284190 29516 ts_tablet_manager.cc:1367] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240502 08:19:53.284768 29516 raft_consensus.cc:348] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:53.284960 29516 raft_consensus.cc:729] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Initialized, Role: FOLLOWER
I20240502 08:19:53.285149 29516 consensus_queue.cc:260] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } }
I20240502 08:19:53.287714 29514 tablet_copy_client.cc:783] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:53.287900 29514 tablet_copy_client.cc:647] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:53.292322 29516 ts_tablet_manager.cc:1398] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 08:19:53.292909 15775 tablet_copy_service.cc:342] P 9333cb10227b4801b4fa7c24e22e131b: Request end of tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-aadd5844b106485e95ecf6fa923cd6ea received from {username='slave'} at 127.6.5.4:37117
I20240502 08:19:53.293006 15775 tablet_copy_service.cc:433] P 9333cb10227b4801b4fa7c24e22e131b: ending tablet copy session 6a0b05f3250d415590b0edf6b44fbe1f-aadd5844b106485e95ecf6fa923cd6ea on tablet aadd5844b106485e95ecf6fa923cd6ea with peer 6a0b05f3250d415590b0edf6b44fbe1f
I20240502 08:19:53.294294 29514 tablet_copy_client.cc:523] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f84bb7d7a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 08:19:53.300920 29514 tablet_bootstrap.cc:492] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee: Bootstrap starting.
@ 0x7f84bb7ba981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f84bb7ba70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 08:19:53.319027 29514 tablet_bootstrap.cc:492] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:53.319281 29514 tablet_bootstrap.cc:492] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee: Bootstrap complete.
I20240502 08:19:53.319495 29514 ts_tablet_manager.cc:1367] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20240502 08:19:53.320108 29514 raft_consensus.cc:348] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:53.320307 29514 raft_consensus.cc:729] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Initialized, Role: FOLLOWER
I20240502 08:19:53.320530 29514 consensus_queue.cc:260] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:53.321151 29514 ts_tablet_manager.cc:1398] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:19:53.322181 15907 tablet_copy_service.cc:342] P 10f0e36203914b85b77e78609ffced98: Request end of tablet copy session 29b035434087435cbaf4ebcac98eedee-01b4100f5ab3405cabd93cff75f61822 received from {username='slave'} at 127.6.5.1:34995
I20240502 08:19:53.322278 15907 tablet_copy_service.cc:433] P 10f0e36203914b85b77e78609ffced98: ending tablet copy session 29b035434087435cbaf4ebcac98eedee-01b4100f5ab3405cabd93cff75f61822 on tablet 01b4100f5ab3405cabd93cff75f61822 with peer 29b035434087435cbaf4ebcac98eedee
@ 0x7f84ac213b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f84ac21719c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f84ac217019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f84ac1f201e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f84af5776db start_thread at ??:0
@ 0x7f84a83b571f clone at ??:0
I20240502 08:19:53.368230 13119 raft_consensus.cc:1203] T 495ea7de81d34b50a92f6cb52795fadf P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:53.524910 12727 raft_consensus.cc:1203] T 01b4100f5ab3405cabd93cff75f61822 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:53.542101 13119 raft_consensus.cc:1203] T aadd5844b106485e95ecf6fa923cd6ea P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 08:19:53.776454 15865 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 08:19:53.788909 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:19:53.798193 13058 connection.cc:535] client connection to 127.6.5.6:44983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:53.798449 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:53.800101 12666 connection.cc:535] client connection to 127.6.5.6:44983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:53.800309 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:19:53.807473 12970 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240502 08:19:53.825080 15822 connection.cc:535] client connection to 127.6.5.6:44983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:53.825294 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:19:53.830374 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:19:53.830694 12798 connection.cc:535] client connection to 127.6.5.6:44983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:53.830933 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:53.860266 15421 connection.cc:535] client connection to 127.6.5.6:44983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:53.860517 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:53.861968 12928 connection.cc:535] client connection to 127.6.5.6:44983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:53.862267 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:53.869374 29606 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:19:53.799481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.5.6:44983 (local address 127.6.5.4:57971)
0502 08:19:53.799825 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0502 08:19:53.799843 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 08:19:53.799968 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:19:53.868860 (+ 68892us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":183,"threads_started":1}
W20240502 08:19:53.869920 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.870030 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.883694 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.900410 29616 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:19:53.848472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.5.6:44983 (local address 127.6.5.8:40281)
0502 08:19:53.899317 (+ 50845us) negotiation.cc:107] Waiting for socket to connect
0502 08:19:53.899418 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":50756,"thread_start_us":48000,"threads_started":1}
W20240502 08:19:53.902166 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.902336 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.902416 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.903276 12798 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.6:44983, user_credentials={real_user=slave}} blocked reactor thread for 72391us
W20240502 08:19:53.906029 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.906473 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.906708 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.906806 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.906878 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.906952 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.907016 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.907094 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.907572 15690 connection.cc:535] client connection to 127.6.5.6:44983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:53.908325 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:53.908689 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.909106 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.909201 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.909269 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.909322 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.913411 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.913697 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.913865 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.913996 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.914433 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.914677 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.914798 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:19:53.915417 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.924688 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.924933 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.925068 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.925190 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.925628 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.925781 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.926595 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.934556 12666 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.6:44983, user_credentials={real_user=slave}} blocked reactor thread for 134276us
I20240502 08:19:53.939276 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:19:53.952627 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.952766 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.952840 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.953014 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:53.954836 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:53.977558 15468 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 08:19:54.027220 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.055717 12842 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 08:19:54.216846 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(660b389a179f49eeb3007dea6ce078e9): perf score=0.033373
I20240502 08:19:54.221643 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(7a8e5440619a4cb0b40dd286862184cd): perf score=0.033380
I20240502 08:19:54.231164 29450 raft_consensus.cc:482] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.231312 29450 raft_consensus.cc:504] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:54.231983 29450 leader_election.cc:288] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
W20240502 08:19:54.233661 15690 leader_election.cc:334] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.236297 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb73ec8c37464c8ba97ac2372667525a" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:19:54.237010 15688 leader_election.cc:302] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:54.239629 29450 raft_consensus.cc:2726] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:54.277266 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(6b40f3044cd6485ca72b05873dc93f7e): perf score=0.033387
I20240502 08:19:54.284592 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(660b389a179f49eeb3007dea6ce078e9): perf score=0.033390
I20240502 08:19:54.322141 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(7a8e5440619a4cb0b40dd286862184cd) complete. Timing: real 0.100s user 0.039s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":12426,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":10578,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":909,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":32349,"threads_started":2,"wal-append.queue_time_us":19899}
I20240502 08:19:54.324302 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(6b40f3044cd6485ca72b05873dc93f7e): perf score=0.033405
I20240502 08:19:54.332828 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(660b389a179f49eeb3007dea6ce078e9) complete. Timing: real 0.116s user 0.024s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6440,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1951,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":953,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":18328,"threads_started":2,"wal-append.queue_time_us":12053}
I20240502 08:19:54.335557 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(660b389a179f49eeb3007dea6ce078e9): 4586 bytes on disk
I20240502 08:19:54.337596 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(660b389a179f49eeb3007dea6ce078e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240502 08:19:54.338135 29669 raft_consensus.cc:482] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.338281 29669 raft_consensus.cc:504] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:54.339249 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcbd19e0a9b49eaa3e403683fe6b34b" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:19:54.339428 29669 leader_election.cc:288] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
W20240502 08:19:54.342341 12928 leader_election.cc:334] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.357898 12927 leader_election.cc:302] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4; no voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.358230 29662 raft_consensus.cc:2726] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:54.364840 29660 raft_consensus.cc:482] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.364993 29660 raft_consensus.cc:504] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:54.365593 29660 leader_election.cc:288] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.366685 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb73ec8c37464c8ba97ac2372667525a" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:19:54.366894 15755 raft_consensus.cc:2454] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:19:54.367986 12795 leader_election.cc:302] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters:
W20240502 08:19:54.368997 12798 leader_election.cc:334] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.369328 29660 raft_consensus.cc:2781] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.369423 29660 raft_consensus.cc:482] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.369514 29660 raft_consensus.cc:3037] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.372071 29660 raft_consensus.cc:504] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:54.375001 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb73ec8c37464c8ba97ac2372667525a" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:19:54.375164 15755 raft_consensus.cc:3037] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.377910 15755 raft_consensus.cc:2454] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 2.
I20240502 08:19:54.380151 12795 leader_election.cc:302] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:19:54.380416 29170 raft_consensus.cc:2781] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Leader election won for term 2
W20240502 08:19:54.383716 12798 leader_election.cc:334] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.384047 29660 leader_election.cc:288] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.403331 29170 raft_consensus.cc:686] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [term 2 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:19:54.403609 29170 consensus_queue.cc:237] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:54.426041 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(6b40f3044cd6485ca72b05873dc93f7e) complete. Timing: real 0.148s user 0.032s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":273,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":36571,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":676,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":1792,"thread_start_us":40907,"threads_started":2,"wal-append.queue_time_us":29122}
I20240502 08:19:54.428833 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(7a8e5440619a4cb0b40dd286862184cd): perf score=0.033427
I20240502 08:19:54.434325 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(660b389a179f49eeb3007dea6ce078e9) complete. Timing: real 0.150s user 0.036s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4733,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":2785,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":886,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":38909,"threads_started":2,"wal-append.queue_time_us":30383}
I20240502 08:19:54.436865 29172 raft_consensus.cc:482] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.437013 29172 raft_consensus.cc:504] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:54.437608 29172 leader_election.cc:288] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:54.437829 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(660b389a179f49eeb3007dea6ce078e9): 4586 bytes on disk
I20240502 08:19:54.438876 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204315ebcf4ccbb0a62de8daa5cb23" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
W20240502 08:19:54.440541 15421 leader_election.cc:334] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.440649 15421 leader_election.cc:302] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db7893bb29a047b3b82cc60bfd8930a3; no voters: 60b6e265e7754b7187307d05ef405dd4, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.441184 29450 raft_consensus.cc:482] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.441314 28613 raft_consensus.cc:2726] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:54.441324 29450 raft_consensus.cc:504] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:54.441924 29450 leader_election.cc:288] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.442797 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9081ea0b98443d38f42c14cd626f0d0" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:19:54.444324 12586 catalog_manager.cc:5484] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 2 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
W20240502 08:19:54.445999 15690 leader_election.cc:334] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.446502 15690 leader_election.cc:302] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:54.447290 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(660b389a179f49eeb3007dea6ce078e9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":708,"lbm_reads_lt_1ms":4}
I20240502 08:19:54.450543 29172 raft_consensus.cc:482] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.451264 29450 raft_consensus.cc:2726] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:54.451100 29172 raft_consensus.cc:504] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:54.454146 29172 leader_election.cc:288] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.454672 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c58721561064e97909a0caddc47a1f0" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
W20240502 08:19:54.456449 15421 leader_election.cc:334] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.456596 15421 leader_election.cc:302] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db7893bb29a047b3b82cc60bfd8930a3; no voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.456997 29172 raft_consensus.cc:2726] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:54.466925 29450 raft_consensus.cc:482] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.467082 29450 raft_consensus.cc:504] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:54.467784 29450 leader_election.cc:288] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:19:54.468281 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e9a523fee46a08a4c6180058ed923" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
W20240502 08:19:54.469120 15690 leader_election.cc:334] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.469224 15690 leader_election.cc:302] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b; no voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.469843 28449 raft_consensus.cc:2726] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:54.470445 29662 raft_consensus.cc:482] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.470642 29662 raft_consensus.cc:504] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:54.471222 29662 leader_election.cc:288] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.471796 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204315ebcf4ccbb0a62de8daa5cb23" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:19:54.471992 15488 raft_consensus.cc:2454] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:19:54.474870 29412 raft_consensus.cc:482] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.474964 12926 leader_election.cc:302] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:19:54.475018 29412 raft_consensus.cc:504] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:54.475643 29412 leader_election.cc:288] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
W20240502 08:19:54.476655 12928 leader_election.cc:334] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.477052 29662 raft_consensus.cc:2781] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.477140 29662 raft_consensus.cc:482] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.477245 29662 raft_consensus.cc:3037] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:19:54.477239 15822 leader_election.cc:334] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.477617 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e9a523fee46a08a4c6180058ed923" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:19:54.477777 15755 raft_consensus.cc:2454] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:19:54.478135 15818 leader_election.cc:302] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.478514 29412 raft_consensus.cc:2781] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.478601 29412 raft_consensus.cc:482] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.478686 29412 raft_consensus.cc:3037] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.480665 29662 raft_consensus.cc:504] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:54.481599 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84204315ebcf4ccbb0a62de8daa5cb23" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:19:54.481786 15488 raft_consensus.cc:3037] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.481873 29450 raft_consensus.cc:482] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.481993 29450 raft_consensus.cc:504] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:54.482882 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e51af7112424e52a9e9c3200a30f86d" candidate_uuid: "9333cb10227b4801b4fa7c24e22e131b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:19:54.483418 29450 leader_election.cc:288] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.484592 15488 raft_consensus.cc:2454] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 2.
W20240502 08:19:54.484740 15690 leader_election.cc:334] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.484839 15690 leader_election.cc:302] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b; no voters: 10f0e36203914b85b77e78609ffced98, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.485229 29450 raft_consensus.cc:2726] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:54.485492 12926 leader_election.cc:302] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, db7893bb29a047b3b82cc60bfd8930a3; no voters:
I20240502 08:19:54.485738 29669 raft_consensus.cc:2781] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:54.485873 29669 raft_consensus.cc:686] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [term 2 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
I20240502 08:19:54.486136 29669 consensus_queue.cc:237] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:54.481173 29412 raft_consensus.cc:504] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:54.487330 29412 leader_election.cc:288] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
W20240502 08:19:54.489200 15822 leader_election.cc:334] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.491909 12580 catalog_manager.cc:5484] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 2 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
W20240502 08:19:54.492872 12928 leader_election.cc:334] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.495692 29662 leader_election.cc:288] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.502666 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188e9a523fee46a08a4c6180058ed923" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:19:54.502830 15755 raft_consensus.cc:3037] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.505461 15755 raft_consensus.cc:2454] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 2.
I20240502 08:19:54.507779 15818 leader_election.cc:302] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.508029 29412 raft_consensus.cc:2781] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:54.508308 29412 raft_consensus.cc:686] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [term 2 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:19:54.508564 29412 consensus_queue.cc:237] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } }
I20240502 08:19:54.511653 29267 raft_consensus.cc:482] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.511811 29267 raft_consensus.cc:504] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:54.512526 29267 leader_election.cc:288] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:19:54.513051 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4a5f0c5c184b7f975f7bcc59cb6b4c" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
W20240502 08:19:54.513852 12666 leader_election.cc:334] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.513991 12666 leader_election.cc:302] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, db7893bb29a047b3b82cc60bfd8930a3
I20240502 08:19:54.514256 29130 raft_consensus.cc:482] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.514382 29267 raft_consensus.cc:2726] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:54.514426 29130 raft_consensus.cc:504] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:54.516103 29130 leader_election.cc:288] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:54.517021 12992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7082603ad2d6450e8863cd53b1897b2e" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:19:54.519069 12581 catalog_manager.cc:5484] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 2 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } }
W20240502 08:19:54.523031 13058 leader_election.cc:334] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.523157 13058 leader_election.cc:302] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f; no voters: 60b6e265e7754b7187307d05ef405dd4, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.523561 29130 raft_consensus.cc:2726] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:54.539564 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(6b40f3044cd6485ca72b05873dc93f7e) complete. Timing: real 0.215s user 0.059s sys 0.023s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":1050,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":879,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":98084,"threads_started":1,"wal-append.queue_time_us":97088}
I20240502 08:19:54.541698 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(7a8e5440619a4cb0b40dd286862184cd): 4925 bytes on disk
I20240502 08:19:54.542137 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(7a8e5440619a4cb0b40dd286862184cd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 08:19:54.544018 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(6b40f3044cd6485ca72b05873dc93f7e): 4840 bytes on disk
I20240502 08:19:54.544531 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(6b40f3044cd6485ca72b05873dc93f7e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 08:19:54.553663 29172 raft_consensus.cc:482] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.553817 29172 raft_consensus.cc:504] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:54.554579 29172 leader_election.cc:288] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:54.558390 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4a5f0c5c184b7f975f7bcc59cb6b4c" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:19:54.558630 12727 raft_consensus.cc:2454] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 1.
W20240502 08:19:54.558784 15421 leader_election.cc:334] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:19:54.559026 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.559067 15419 leader_election.cc:302] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.559499 29172 raft_consensus.cc:2781] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.559594 29172 raft_consensus.cc:482] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.559686 29172 raft_consensus.cc:3037] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.563320 29172 raft_consensus.cc:504] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:54.564939 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4a5f0c5c184b7f975f7bcc59cb6b4c" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee"
I20240502 08:19:54.565140 12727 raft_consensus.cc:3037] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:19:54.566365 15421 leader_election.cc:334] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.567142 29691 raft_consensus.cc:482] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.567322 29691 raft_consensus.cc:504] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:54.568392 29691 leader_election.cc:288] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:19:54.568522 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de012363e4f4455acd96cb674ab0c4b" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
W20240502 08:19:54.569275 15822 leader_election.cc:334] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.569403 15822 leader_election.cc:302] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10f0e36203914b85b77e78609ffced98; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:54.569667 29691 raft_consensus.cc:2726] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:54.569412 12727 raft_consensus.cc:2454] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7893bb29a047b3b82cc60bfd8930a3 in term 2.
I20240502 08:19:54.570154 15419 leader_election.cc:302] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29b035434087435cbaf4ebcac98eedee, db7893bb29a047b3b82cc60bfd8930a3; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.570452 28613 raft_consensus.cc:2781] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:54.570454 29172 leader_election.cc:288] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:54.571240 28613 raft_consensus.cc:686] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [term 2 LEADER]: Becoming Leader. State: Replica: db7893bb29a047b3b82cc60bfd8930a3, State: Running, Role: LEADER
I20240502 08:19:54.571534 28613 consensus_queue.cc:237] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:54.581043 12581 catalog_manager.cc:5484] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5). New cstate: current_term: 2 leader_uuid: "db7893bb29a047b3b82cc60bfd8930a3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:54.584777 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(7a8e5440619a4cb0b40dd286862184cd) complete. Timing: real 0.156s user 0.043s sys 0.031s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1260,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":919,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":23412,"threads_started":1,"wal-append.queue_time_us":23212}
I20240502 08:19:54.587363 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(7a8e5440619a4cb0b40dd286862184cd): 4925 bytes on disk
I20240502 08:19:54.590059 29691 raft_consensus.cc:482] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.590219 29691 raft_consensus.cc:504] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:54.590943 29691 leader_election.cc:288] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 29b035434087435cbaf4ebcac98eedee (127.6.5.1:33183)
I20240502 08:19:54.591472 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6a660632634e3c9abde21dbf84dfc0" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29b035434087435cbaf4ebcac98eedee" is_pre_election: true
I20240502 08:19:54.592784 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(7a8e5440619a4cb0b40dd286862184cd) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240502 08:19:54.593537 15822 leader_election.cc:334] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.593650 15822 leader_election.cc:302] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10f0e36203914b85b77e78609ffced98; no voters: 29b035434087435cbaf4ebcac98eedee, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.594077 29691 raft_consensus.cc:2726] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:54.596223 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(6b40f3044cd6485ca72b05873dc93f7e): 4840 bytes on disk
I20240502 08:19:54.596771 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(6b40f3044cd6485ca72b05873dc93f7e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 08:19:54.616672 15865 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:54.630424 12708 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:54.639257 29412 raft_consensus.cc:482] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.639405 29412 raft_consensus.cc:504] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:54.640031 29412 leader_election.cc:288] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.641654 13102 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:54.641901 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c58721561064e97909a0caddc47a1f0" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
W20240502 08:19:54.641970 15822 leader_election.cc:334] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.642105 15488 raft_consensus.cc:2454] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:19:54.642685 15819 leader_election.cc:302] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.643146 29691 raft_consensus.cc:2781] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.643224 29691 raft_consensus.cc:482] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.643297 29691 raft_consensus.cc:3037] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.645809 29691 raft_consensus.cc:504] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:54.646430 29691 leader_election.cc:288] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 election: Requested vote from peers db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
W20240502 08:19:54.648487 15822 leader_election.cc:334] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.650593 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c58721561064e97909a0caddc47a1f0" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:19:54.650815 15488 raft_consensus.cc:3037] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.653983 15488 raft_consensus.cc:2454] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 2.
I20240502 08:19:54.655115 15819 leader_election.cc:302] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, db7893bb29a047b3b82cc60bfd8930a3; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.655468 29691 raft_consensus.cc:2781] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:54.655700 29691 raft_consensus.cc:686] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [term 2 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:19:54.655961 29691 consensus_queue.cc:237] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:54.657860 12970 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 08:19:54.660298 29561 raft_consensus.cc:482] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.660471 29561 raft_consensus.cc:504] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:54.661101 29561 leader_election.cc:288] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.662226 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6a660632634e3c9abde21dbf84dfc0" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:19:54.662421 15885 raft_consensus.cc:2454] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 1.
I20240502 08:19:54.662845 12665 leader_election.cc:302] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 29b035434087435cbaf4ebcac98eedee; no voters:
I20240502 08:19:54.663146 29561 raft_consensus.cc:2781] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.663228 29561 raft_consensus.cc:482] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.663309 29561 raft_consensus.cc:3037] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:19:54.664284 12666 leader_election.cc:334] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.666040 29561 raft_consensus.cc:504] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
I20240502 08:19:54.667037 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6a660632634e3c9abde21dbf84dfc0" candidate_uuid: "29b035434087435cbaf4ebcac98eedee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:19:54.667227 15885 raft_consensus.cc:3037] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.669202 29172 raft_consensus.cc:482] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.669344 29172 raft_consensus.cc:504] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:54.669875 15885 raft_consensus.cc:2454] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b035434087435cbaf4ebcac98eedee in term 2.
I20240502 08:19:54.670027 29172 leader_election.cc:288] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.670321 12665 leader_election.cc:302] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 29b035434087435cbaf4ebcac98eedee; no voters:
I20240502 08:19:54.670611 29267 raft_consensus.cc:2781] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:54.673802 29412 raft_consensus.cc:482] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.673930 29412 raft_consensus.cc:504] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:54.674510 29412 leader_election.cc:288] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933)
I20240502 08:19:54.675300 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8774c037b901461684d96460e663d17e" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
W20240502 08:19:54.675771 15822 leader_election.cc:334] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.675918 15819 leader_election.cc:302] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10f0e36203914b85b77e78609ffced98; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:54.676388 29690 raft_consensus.cc:2726] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:19:54.677956 15421 leader_election.cc:334] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.678715 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432a6e076bf4401bb81903d91de38a57" candidate_uuid: "db7893bb29a047b3b82cc60bfd8930a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:19:54.679322 15418 leader_election.cc:302] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db7893bb29a047b3b82cc60bfd8930a3; no voters: 6a0b05f3250d415590b0edf6b44fbe1f, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.679745 29172 raft_consensus.cc:2726] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:19:54.684962 29677 raft_consensus.cc:482] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.685110 29677 raft_consensus.cc:504] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:54.685709 29677 leader_election.cc:288] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
W20240502 08:19:54.686255 12666 leader_election.cc:334] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.686766 29561 leader_election.cc:288] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [CANDIDATE]: Term 2 election: Requested vote from peers 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
W20240502 08:19:54.687593 12798 leader_election.cc:334] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.699654 29677 raft_consensus.cc:482] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.699810 29677 raft_consensus.cc:504] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:54.700359 29677 leader_election.cc:288] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
W20240502 08:19:54.702280 12798 leader_election.cc:334] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.703385 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8774c037b901461684d96460e663d17e" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:19:54.703621 15885 raft_consensus.cc:2454] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:19:54.705029 12797 leader_election.cc:302] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.705475 29677 raft_consensus.cc:2781] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.705566 29677 raft_consensus.cc:482] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.705648 29677 raft_consensus.cc:3037] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.708195 29677 raft_consensus.cc:504] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:54.709693 29267 raft_consensus.cc:686] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [term 2 LEADER]: Becoming Leader. State: Replica: 29b035434087435cbaf4ebcac98eedee, State: Running, Role: LEADER
I20240502 08:19:54.710448 29267 consensus_queue.cc:237] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } }
W20240502 08:19:54.711380 12798 leader_election.cc:334] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.711598 29677 leader_election.cc:288] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
I20240502 08:19:54.715093 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8774c037b901461684d96460e663d17e" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:19:54.715413 15885 raft_consensus.cc:3037] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.719009 15885 raft_consensus.cc:2454] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 2.
I20240502 08:19:54.730083 29170 raft_consensus.cc:482] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.730218 29170 raft_consensus.cc:504] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:54.730782 29170 leader_election.cc:288] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
W20240502 08:19:54.732740 12798 leader_election.cc:334] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.735951 29715 raft_consensus.cc:482] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.742990 29715 raft_consensus.cc:504] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:54.743525 29715 leader_election.cc:288] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.737965 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de012363e4f4455acd96cb674ab0c4b" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98" is_pre_election: true
I20240502 08:19:54.743705 15887 raft_consensus.cc:2454] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:19:54.744254 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e51af7112424e52a9e9c3200a30f86d" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:19:54.744478 15755 raft_consensus.cc:2454] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:19:54.744901 15818 leader_election.cc:302] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters:
I20240502 08:19:54.745209 29715 raft_consensus.cc:2781] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 08:19:54.745280 15822 leader_election.cc:334] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.745312 29715 raft_consensus.cc:482] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.745414 29715 raft_consensus.cc:3037] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.746117 12797 leader_election.cc:302] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.746526 29677 raft_consensus.cc:2781] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:54.746747 29677 raft_consensus.cc:686] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [term 2 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:19:54.748093 29715 raft_consensus.cc:504] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:54.748510 29677 consensus_queue.cc:237] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:54.748878 29715 leader_election.cc:288] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 election: Requested vote from peers 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.750831 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e51af7112424e52a9e9c3200a30f86d" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:19:54.751024 15755 raft_consensus.cc:3037] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.754180 15755 raft_consensus.cc:2454] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 2.
W20240502 08:19:54.754935 15822 leader_election.cc:334] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.755549 15818 leader_election.cc:302] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 9333cb10227b4801b4fa7c24e22e131b; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.756460 29690 raft_consensus.cc:2781] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:54.756603 29690 raft_consensus.cc:686] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [term 2 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:19:54.756875 29690 consensus_queue.cc:237] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:54.770591 12581 catalog_manager.cc:5484] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to 29b035434087435cbaf4ebcac98eedee (127.6.5.1). New cstate: current_term: 2 leader_uuid: "29b035434087435cbaf4ebcac98eedee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 } } }
I20240502 08:19:54.773272 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9081ea0b98443d38f42c14cd626f0d0" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b" is_pre_election: true
I20240502 08:19:54.773463 15755 raft_consensus.cc:2454] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:19:54.774278 12795 leader_election.cc:302] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.774691 29170 raft_consensus.cc:2781] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.774776 29170 raft_consensus.cc:482] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.774848 29170 raft_consensus.cc:3037] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.777539 29170 raft_consensus.cc:504] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:54.778976 29690 raft_consensus.cc:482] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.779124 29690 raft_consensus.cc:504] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:54.779273 29662 raft_consensus.cc:482] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.779417 29662 raft_consensus.cc:504] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:54.779803 29690 leader_election.cc:288] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
I20240502 08:19:54.779978 29662 leader_election.cc:288] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.780875 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7082603ad2d6450e8863cd53b1897b2e" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:19:54.781093 13119 raft_consensus.cc:2454] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 1.
I20240502 08:19:54.781085 29660 raft_consensus.cc:482] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.781232 29660 raft_consensus.cc:504] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:54.781478 12925 leader_election.cc:302] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
W20240502 08:19:54.781530 12798 leader_election.cc:334] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.781800 29660 leader_election.cc:288] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.781867 29662 raft_consensus.cc:2781] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.781955 29662 raft_consensus.cc:482] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.782044 29662 raft_consensus.cc:3037] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.782141 29170 leader_election.cc:288] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 9333cb10227b4801b4fa7c24e22e131b (127.6.5.7:38851)
W20240502 08:19:54.784149 12928 leader_election.cc:334] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:19:54.784516 15822 leader_election.cc:334] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.784777 29662 raft_consensus.cc:504] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:54.785567 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7082603ad2d6450e8863cd53b1897b2e" candidate_uuid: "60b6e265e7754b7187307d05ef405dd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:19:54.785737 13119 raft_consensus.cc:3037] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.785807 29662 leader_election.cc:288] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.788254 13119 raft_consensus.cc:2454] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b6e265e7754b7187307d05ef405dd4 in term 2.
I20240502 08:19:54.788622 29130 raft_consensus.cc:482] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.788725 29130 raft_consensus.cc:504] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:54.789266 29130 leader_election.cc:288] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
I20240502 08:19:54.790046 12925 leader_election.cc:302] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b6e265e7754b7187307d05ef405dd4, 6a0b05f3250d415590b0edf6b44fbe1f; no voters:
W20240502 08:19:54.790729 12798 leader_election.cc:334] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.791183 12797 leader_election.cc:302] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.791579 29170 raft_consensus.cc:2781] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.791676 29170 raft_consensus.cc:482] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.791766 29170 raft_consensus.cc:3037] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.792161 29669 raft_consensus.cc:2781] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:54.792284 29669 raft_consensus.cc:686] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [term 2 LEADER]: Becoming Leader. State: Replica: 60b6e265e7754b7187307d05ef405dd4, State: Running, Role: LEADER
W20240502 08:19:54.792474 13058 leader_election.cc:334] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.792505 29669 consensus_queue.cc:237] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
W20240502 08:19:54.793851 12928 leader_election.cc:334] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.794535 29170 raft_consensus.cc:504] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:54.795240 29170 leader_election.cc:288] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 10f0e36203914b85b77e78609ffced98 (127.6.5.8:46303)
W20240502 08:19:54.802974 12798 leader_election.cc:334] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.806135 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcbd19e0a9b49eaa3e403683fe6b34b" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4" is_pre_election: true
I20240502 08:19:54.806324 12990 raft_consensus.cc:2454] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 1.
I20240502 08:19:54.807483 15819 leader_election.cc:302] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 60b6e265e7754b7187307d05ef405dd4; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.807730 29690 raft_consensus.cc:2781] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.807828 29690 raft_consensus.cc:482] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.807924 29690 raft_consensus.cc:3037] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.808140 15887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de012363e4f4455acd96cb674ab0c4b" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10f0e36203914b85b77e78609ffced98"
I20240502 08:19:54.808311 15887 raft_consensus.cc:3037] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.810359 29690 raft_consensus.cc:504] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:54.810652 15887 raft_consensus.cc:2454] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 2.
I20240502 08:19:54.812336 29690 leader_election.cc:288] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3:46757)
W20240502 08:19:54.812506 15822 leader_election.cc:334] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.819514 12581 catalog_manager.cc:5484] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to 60b6e265e7754b7187307d05ef405dd4 (127.6.5.3). New cstate: current_term: 2 leader_uuid: "60b6e265e7754b7187307d05ef405dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
W20240502 08:19:54.823609 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:54.826050 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9081ea0b98443d38f42c14cd626f0d0" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9333cb10227b4801b4fa7c24e22e131b"
I20240502 08:19:54.826222 15755 raft_consensus.cc:3037] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.829084 15755 raft_consensus.cc:2454] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 2.
I20240502 08:19:54.829589 12795 leader_election.cc:302] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9333cb10227b4801b4fa7c24e22e131b, e1dd14f7ca764a1490df419f13f2804a; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.829986 29170 raft_consensus.cc:2781] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:54.830981 29170 raft_consensus.cc:686] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [term 2 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:19:54.831235 29170 consensus_queue.cc:237] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } }
I20240502 08:19:54.834022 12797 leader_election.cc:302] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, e1dd14f7ca764a1490df419f13f2804a; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.834832 12990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcbd19e0a9b49eaa3e403683fe6b34b" candidate_uuid: "10f0e36203914b85b77e78609ffced98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60b6e265e7754b7187307d05ef405dd4"
I20240502 08:19:54.835031 12990 raft_consensus.cc:3037] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.837697 12990 raft_consensus.cc:2454] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f0e36203914b85b77e78609ffced98 in term 2.
I20240502 08:19:54.838507 15819 leader_election.cc:302] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10f0e36203914b85b77e78609ffced98, 60b6e265e7754b7187307d05ef405dd4; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.838769 29690 raft_consensus.cc:2781] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:54.838910 29690 raft_consensus.cc:686] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [term 2 LEADER]: Becoming Leader. State: Replica: 10f0e36203914b85b77e78609ffced98, State: Running, Role: LEADER
I20240502 08:19:54.839161 29690 consensus_queue.cc:237] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } }
I20240502 08:19:54.841804 12581 catalog_manager.cc:5484] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 2 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:54.849169 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:19:54.849906 29170 raft_consensus.cc:2781] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:54.854377 12581 catalog_manager.cc:5484] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 2 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } }
I20240502 08:19:54.858011 28924 raft_consensus.cc:482] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.858148 28924 raft_consensus.cc:504] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:54.858798 28924 leader_election.cc:288] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
W20240502 08:19:54.860327 13058 leader_election.cc:334] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.861199 15755 raft_consensus.cc:1261] T bb73ec8c37464c8ba97ac2372667525a P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:19:54.864145 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5af24c28aa04f72844744771f6b7249" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" is_pre_election: true
I20240502 08:19:54.864317 13119 raft_consensus.cc:2454] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dd14f7ca764a1490df419f13f2804a in term 1.
I20240502 08:19:54.865376 29660 consensus_queue.cc:1035] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:54.865736 29170 raft_consensus.cc:686] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [term 2 LEADER]: Becoming Leader. State: Replica: e1dd14f7ca764a1490df419f13f2804a, State: Running, Role: LEADER
I20240502 08:19:54.866058 29170 consensus_queue.cc:237] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } }
I20240502 08:19:54.881472 12860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5af24c28aa04f72844744771f6b7249" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:19:54.881665 12860 raft_consensus.cc:2454] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
I20240502 08:19:54.882529 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432a6e076bf4401bb81903d91de38a57" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3" is_pre_election: true
I20240502 08:19:54.882817 15487 raft_consensus.cc:2454] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 1.
W20240502 08:19:54.887230 15819 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 48751.3us
I20240502 08:19:54.895515 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:54.904348 13056 leader_election.cc:302] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, db7893bb29a047b3b82cc60bfd8930a3; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.906769 28924 raft_consensus.cc:2781] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.906878 28924 raft_consensus.cc:482] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.906963 28924 raft_consensus.cc:3037] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:19:54.908742 12795 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.7:38851, user_credentials={real_user=slave}} blocked reactor thread for 47129.8us
I20240502 08:19:54.909173 12795 leader_election.cc:302] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.909204 12581 catalog_manager.cc:5484] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to e1dd14f7ca764a1490df419f13f2804a (127.6.5.2). New cstate: current_term: 2 leader_uuid: "e1dd14f7ca764a1490df419f13f2804a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } }
I20240502 08:19:54.909741 28924 raft_consensus.cc:504] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
I20240502 08:19:54.911520 29660 raft_consensus.cc:2781] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.911614 29660 raft_consensus.cc:482] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.911703 29660 raft_consensus.cc:3037] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.915927 29660 raft_consensus.cc:504] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:54.916594 29660 leader_election.cc:288] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Requested vote from peers 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4:32979), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:54.910363 28924 leader_election.cc:288] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 election: Requested vote from peers 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983), db7893bb29a047b3b82cc60bfd8930a3 (127.6.5.5:44131)
W20240502 08:19:54.926077 13058 leader_election.cc:334] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.933377 15755 raft_consensus.cc:1261] T 188e9a523fee46a08a4c6180058ed923 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:54.933964 29715 consensus_queue.cc:1035] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:19:54.954960 13056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.2:36933, user_credentials={real_user=slave}} blocked reactor thread for 50070.1us
I20240502 08:19:54.955207 13056 leader_election.cc:302] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:54.956269 29130 raft_consensus.cc:2781] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:19:54.956383 29130 raft_consensus.cc:482] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d45ab45b8eb4d03a4bb9cb88b78a98b)
I20240502 08:19:54.956478 29130 raft_consensus.cc:3037] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.959254 29130 raft_consensus.cc:504] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:54.959859 29130 leader_election.cc:288] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
W20240502 08:19:54.962954 13058 leader_election.cc:334] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:54.983695 15487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432a6e076bf4401bb81903d91de38a57" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db7893bb29a047b3b82cc60bfd8930a3"
I20240502 08:19:54.983886 15487 raft_consensus.cc:3037] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:54.987430 13119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5af24c28aa04f72844744771f6b7249" candidate_uuid: "e1dd14f7ca764a1490df419f13f2804a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a0b05f3250d415590b0edf6b44fbe1f"
I20240502 08:19:54.987679 13119 raft_consensus.cc:2379] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e1dd14f7ca764a1490df419f13f2804a in current term 2: Already voted for candidate 6a0b05f3250d415590b0edf6b44fbe1f in this term.
W20240502 08:19:54.992660 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:54.996699 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:55.001910 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5af24c28aa04f72844744771f6b7249" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:19:55.002189 12862 raft_consensus.cc:2379] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6a0b05f3250d415590b0edf6b44fbe1f in current term 2: Already voted for candidate e1dd14f7ca764a1490df419f13f2804a in this term.
I20240502 08:19:55.002380 15487 raft_consensus.cc:2454] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 2.
I20240502 08:19:55.003351 13056 leader_election.cc:302] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b, e1dd14f7ca764a1490df419f13f2804a
I20240502 08:19:55.006116 29728 raft_consensus.cc:2726] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:19:55.006551 13056 leader_election.cc:302] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, db7893bb29a047b3b82cc60bfd8930a3; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:55.007133 29728 raft_consensus.cc:2781] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:55.007302 29728 raft_consensus.cc:686] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:19:55.007632 29728 consensus_queue.cc:237] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } }
W20240502 08:19:55.012511 12798 leader_election.cc:334] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:55.013406 12798 leader_election.cc:302] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1dd14f7ca764a1490df419f13f2804a; no voters: 6a0b05f3250d415590b0edf6b44fbe1f, 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:55.013957 29726 raft_consensus.cc:2726] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240502 08:19:55.023281 15807 kernel_stack_watchdog.cc:198] Thread 29691 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 289ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f81c3800980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f81c046ed11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f81c048f7ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f81c048b3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f81c04904ff kudu::ThreadPool::Submit() at ??:0
@ 0x7f81cb62fe87 kudu::log::Log::AppendThread::Wake() at ??:0
@ 0x7f81cb63cfbc kudu::log::Log::AsyncAppend() at ??:0
@ 0x7f81cb63d995 kudu::log::Log::AsyncAppendReplicates() at ??:0
@ 0x7f81cbc3b47d kudu::consensus::LogCache::AppendOperations() at ??:0
@ 0x7f81cbbdcb1b kudu::consensus::PeerMessageQueue::AppendOperations() at ??:0
@ 0x7f81cbbdc12f kudu::consensus::PeerMessageQueue::AppendOperation() at ??:0
@ 0x7f81cbc8f2d6 kudu::consensus::RaftConsensus::AppendNewRoundToQueueUnlocked() at ??:0
@ 0x7f81cbc86ad4 kudu::consensus::RaftConsensus::BecomeLeaderUnlocked() at ??:0
W20240502 08:19:55.024554 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:55.029659 12580 catalog_manager.cc:5484] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 2 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } }
I20240502 08:19:55.031956 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:55.034754 15487 raft_consensus.cc:1261] T 84204315ebcf4ccbb0a62de8daa5cb23 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:19:55.035403 29662 consensus_queue.cc:1035] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:55.038769 12581 catalog_manager.cc:5484] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 2 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 } } peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:55.049707 12580 catalog_manager.cc:5484] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 2 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:55.050125 12580 catalog_manager.cc:5484] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 reported cstate change: term changed from 1 to 2, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to 10f0e36203914b85b77e78609ffced98 (127.6.5.8). New cstate: current_term: 2 leader_uuid: "10f0e36203914b85b77e78609ffced98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } peers { permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 } } }
W20240502 08:19:55.095325 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:55.114200 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:55.137785 12730 raft_consensus.cc:1261] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P 29b035434087435cbaf4ebcac98eedee [term 2 FOLLOWER]: Refusing update from remote peer db7893bb29a047b3b82cc60bfd8930a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:55.138613 28613 consensus_queue.cc:1035] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b035434087435cbaf4ebcac98eedee" member_type: VOTER last_known_addr { host: "127.6.5.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:55.171257 15887 raft_consensus.cc:1261] T 4e6a660632634e3c9abde21dbf84dfc0 P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Refusing update from remote peer 29b035434087435cbaf4ebcac98eedee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:19:55.172120 29713 consensus_queue.cc:1035] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:55.174634 13119 raft_consensus.cc:1261] T 7082603ad2d6450e8863cd53b1897b2e P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Refusing update from remote peer 60b6e265e7754b7187307d05ef405dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:55.175176 29662 consensus_queue.cc:1035] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:19:55.182361 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:55.190030 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:55.231338 12990 raft_consensus.cc:1261] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 60b6e265e7754b7187307d05ef405dd4 [term 2 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:55.231855 29691 consensus_queue.cc:1035] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b6e265e7754b7187307d05ef405dd4" member_type: VOTER last_known_addr { host: "127.6.5.3" port: 46757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:19:55.247416 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:55.267210 15755 raft_consensus.cc:1261] T e9081ea0b98443d38f42c14cd626f0d0 P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:55.267663 29170 consensus_queue.cc:1035] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:19:55.273680 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:55.278223 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:55.283133 15487 raft_consensus.cc:1261] T 3c58721561064e97909a0caddc47a1f0 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:19:55.283615 29690 consensus_queue.cc:1035] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:55.308174 15885 raft_consensus.cc:1261] T 8774c037b901461684d96460e663d17e P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:55.308686 29726 consensus_queue.cc:1035] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:55.313740 15755 raft_consensus.cc:1261] T 9e51af7112424e52a9e9c3200a30f86d P 9333cb10227b4801b4fa7c24e22e131b [term 2 FOLLOWER]: Refusing update from remote peer 10f0e36203914b85b77e78609ffced98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:19:55.314620 29691 consensus_queue.cc:1035] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9333cb10227b4801b4fa7c24e22e131b" member_type: VOTER last_known_addr { host: "127.6.5.7" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:19:55.320745 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:55.334861 15885 raft_consensus.cc:1261] T 8de012363e4f4455acd96cb674ab0c4b P 10f0e36203914b85b77e78609ffced98 [term 2 FOLLOWER]: Refusing update from remote peer e1dd14f7ca764a1490df419f13f2804a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:55.335330 29726 consensus_queue.cc:1035] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f0e36203914b85b77e78609ffced98" member_type: VOTER last_known_addr { host: "127.6.5.8" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:19:55.362967 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:19:55.556722 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:55.559937 15487 raft_consensus.cc:1261] T 432a6e076bf4401bb81903d91de38a57 P db7893bb29a047b3b82cc60bfd8930a3 [term 2 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:19:55.560642 29728 consensus_queue.cc:1035] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7893bb29a047b3b82cc60bfd8930a3" member_type: VOTER last_known_addr { host: "127.6.5.5" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 08:19:55.630941 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:55.706671 13102 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 08:19:55.707130 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:19:55.708914 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:19:55.730707 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:19:55.781440 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:55.789849 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:55.794124 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:55.799401 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:19:55.884840 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:55.937381 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:55.948575 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.024197 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.026703 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.042068 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.043781 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.065436 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.072463 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.072996 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.080263 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.099099 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.120376 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.123229 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.137590 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.150265 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:56.155365 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:56.158829 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:19:56.175674 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:56.180856 13102 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 08:19:56.189606 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:56.192219 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:19:56.203121 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:19:56.214449 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.214972 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:56.248551 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:19:56.250932 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:56.252774 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:56.260569 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:19:56.277226 15419 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.2:36933, user_credentials={real_user=slave}} blocked reactor thread for 104292us
W20240502 08:19:56.305104 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.305372 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.328470 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.346843 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.352965 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.374320 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.383646 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.424191 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.506067 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.523049 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.551726 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.571979 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.572566 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.598659 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.615183 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:56.647177 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:19:56.660321 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:56.884876 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:56.895154 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:19:56.906800 29717 raft_consensus.cc:482] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:56.906988 29717 raft_consensus.cc:504] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:56.907677 29717 leader_election.cc:288] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:56.908213 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5af24c28aa04f72844744771f6b7249" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a" is_pre_election: true
I20240502 08:19:56.908418 12862 raft_consensus.cc:2454] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 2.
I20240502 08:19:56.908871 13056 leader_election.cc:302] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters:
I20240502 08:19:56.909148 29717 raft_consensus.cc:2781] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 08:19:56.909240 29717 raft_consensus.cc:482] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:56.909315 29717 raft_consensus.cc:3037] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:19:56.912153 29717 raft_consensus.cc:504] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:56.913908 29717 leader_election.cc:288] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 3 election: Requested vote from peers e1dd14f7ca764a1490df419f13f2804a (127.6.5.2:36933), 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983)
I20240502 08:19:56.914441 12862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5af24c28aa04f72844744771f6b7249" candidate_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e1dd14f7ca764a1490df419f13f2804a"
I20240502 08:19:56.914688 12862 raft_consensus.cc:3037] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 2 FOLLOWER]: Advancing to term 3
W20240502 08:19:56.916034 13058 leader_election.cc:334] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:19:56.916730 13058 leader_election.cc:334] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:56.918629 12862 raft_consensus.cc:2454] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a0b05f3250d415590b0edf6b44fbe1f in term 3.
I20240502 08:19:56.923578 13056 leader_election.cc:302] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a0b05f3250d415590b0edf6b44fbe1f, e1dd14f7ca764a1490df419f13f2804a; no voters: 8d45ab45b8eb4d03a4bb9cb88b78a98b
I20240502 08:19:56.923974 29717 raft_consensus.cc:2781] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:19:56.925812 29717 raft_consensus.cc:686] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [term 3 LEADER]: Becoming Leader. State: Replica: 6a0b05f3250d415590b0edf6b44fbe1f, State: Running, Role: LEADER
I20240502 08:19:56.926096 29717 consensus_queue.cc:237] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } }
I20240502 08:19:56.928345 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:19:56.933418 12579 catalog_manager.cc:5484] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f reported cstate change: term changed from 1 to 3, leader changed from 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6) to 6a0b05f3250d415590b0edf6b44fbe1f (127.6.5.4). New cstate: current_term: 3 leader_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0b05f3250d415590b0edf6b44fbe1f" member_type: VOTER last_known_addr { host: "127.6.5.4" port: 32979 } } peers { permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 } } peers { permanent_uuid: "8d45ab45b8eb4d03a4bb9cb88b78a98b" member_type: VOTER last_known_addr { host: "127.6.5.6" port: 44983 } } }
I20240502 08:19:56.950028 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:56.982578 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:57.026130 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:19:57.046106 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:57.115278 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(1fba813ac35c4382a0917604c2057496): perf score=0.033336
I20240502 08:19:57.122581 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(553cec4eb28341dba84afc79b540ae4b): perf score=0.033339
I20240502 08:19:57.146167 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(1fba813ac35c4382a0917604c2057496) complete. Timing: real 0.031s user 0.022s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":302,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1319,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":512,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2048,"threads_started":2,"wal-append.queue_time_us":1004}
I20240502 08:19:57.148852 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(1fba813ac35c4382a0917604c2057496): 4840 bytes on disk
I20240502 08:19:57.149471 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(1fba813ac35c4382a0917604c2057496) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240502 08:19:57.162118 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:57.171453 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(553cec4eb28341dba84afc79b540ae4b) complete. Timing: real 0.049s user 0.027s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7818,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1152,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1714,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":8161,"threads_started":2,"wal-append.queue_time_us":316}
I20240502 08:19:57.174117 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(553cec4eb28341dba84afc79b540ae4b): 4925 bytes on disk
I20240502 08:19:57.174727 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(553cec4eb28341dba84afc79b540ae4b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 08:19:57.228319 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(553cec4eb28341dba84afc79b540ae4b): perf score=0.033373
I20240502 08:19:57.280272 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(553cec4eb28341dba84afc79b540ae4b) complete. Timing: real 0.051s user 0.026s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10651,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":2988,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":814,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14924,"threads_started":2,"wal-append.queue_time_us":3448}
I20240502 08:19:57.282883 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(553cec4eb28341dba84afc79b540ae4b): 4925 bytes on disk
I20240502 08:19:57.283442 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(553cec4eb28341dba84afc79b540ae4b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 08:19:57.285652 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(e249007ce4284fd2bc026ab9d6c5d4cb): perf score=0.033379
W20240502 08:19:57.310662 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:57.327178 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(e249007ce4284fd2bc026ab9d6c5d4cb): perf score=0.033399
I20240502 08:19:57.347436 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(e249007ce4284fd2bc026ab9d6c5d4cb) complete. Timing: real 0.061s user 0.048s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":248,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1273,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":858,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":235,"threads_started":1,"wal-append.queue_time_us":334}
I20240502 08:19:57.350514 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(e249007ce4284fd2bc026ab9d6c5d4cb): 4586 bytes on disk
I20240502 08:19:57.351222 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(e249007ce4284fd2bc026ab9d6c5d4cb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240502 08:19:57.388312 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:57.399819 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:57.404853 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(e249007ce4284fd2bc026ab9d6c5d4cb) complete. Timing: real 0.077s user 0.025s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5868,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":6434,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":863,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":12454,"threads_started":2,"wal-append.queue_time_us":6388}
I20240502 08:19:57.409857 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(e249007ce4284fd2bc026ab9d6c5d4cb): 4586 bytes on disk
I20240502 08:19:57.410429 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(e249007ce4284fd2bc026ab9d6c5d4cb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240502 08:19:57.418689 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(1fba813ac35c4382a0917604c2057496): perf score=0.033417
W20240502 08:19:57.526057 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:57.539322 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(1fba813ac35c4382a0917604c2057496) complete. Timing: real 0.118s user 0.038s sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1405,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":907,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3017,"threads_started":1,"wal-append.queue_time_us":2925}
I20240502 08:19:57.542076 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(1fba813ac35c4382a0917604c2057496): 4840 bytes on disk
I20240502 08:19:57.542480 12860 raft_consensus.cc:1261] T a5af24c28aa04f72844744771f6b7249 P e1dd14f7ca764a1490df419f13f2804a [term 3 FOLLOWER]: Refusing update from remote peer 6a0b05f3250d415590b0edf6b44fbe1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 08:19:57.543409 30012 consensus_queue.cc:1035] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1dd14f7ca764a1490df419f13f2804a" member_type: VOTER last_known_addr { host: "127.6.5.2" port: 36933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 08:19:57.543674 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(1fba813ac35c4382a0917604c2057496) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 08:19:57.606448 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:57.642347 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:19:57.643296 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:19:57.665309 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:19:57.678411 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:57.688789 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:19:57.697361 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:19:57.731599 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:57.733697 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:19:57.755012 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:57.763815 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:57.779466 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:57.796355 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:57.801826 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:19:57.807109 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:57.837810 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:57.907997 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:57.931699 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:19:58.134967 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:58.170118 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:19:58.173502 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:19:58.186960 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:19:58.197986 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:58.219514 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:19:58.261433 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:19:58.270790 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:19:58.313773 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:19:58.353969 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.368345 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.383786 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.424082 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.458447 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.463881 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.473559 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.494724 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.495214 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.537343 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.548480 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.569882 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.601140 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.631654 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:19:58.643684 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:19:58.662525 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.664017 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:19:58.698964 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:19:58.707692 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:58.708338 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:19:58.720081 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:19:58.729950 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:19:58.730683 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:19:58.744666 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.744787 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.747025 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:19:58.757782 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.765591 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.775425 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.779026 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:19:58.795929 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:19:58.811877 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 08:19:58.821580 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.836710 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 08:19:58.837500 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.843822 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.844869 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 08:19:58.845927 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.866068 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 08:19:58.893549 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.917547 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240502 08:19:58.918438 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.945925 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:58.958122 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 08:19:58.982172 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 08:19:59.004498 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:59.026538 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:19:59.104990 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:19:59.111835 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:19:59.118551 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:19:59.144196 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:19:59.158665 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:59.160578 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:19:59.197052 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:19:59.279934 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:19:59.298440 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:19:59.298910 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:59.309006 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:59.319033 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:19:59.382107 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:59.487535 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:59.745208 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:19:59.842564 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:19:59.860145 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:19:59.890161 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:19:59.905315 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:19:59.905696 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:19:59.908627 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:19:59.923126 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:19:59.939527 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:19:59.958194 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:19:59.958838 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:00.061671 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:20:00.078745 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(f1581261b4c144a9b670786eb3d26466): perf score=0.033334
W20240502 08:20:00.114805 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:20:00.133198 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(f1581261b4c144a9b670786eb3d26466) complete. Timing: real 0.054s user 0.045s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":844,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1321,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":718,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":945,"threads_started":2,"wal-append.queue_time_us":279}
I20240502 08:20:00.136233 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(f1581261b4c144a9b670786eb3d26466): 4586 bytes on disk
I20240502 08:20:00.136801 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(f1581261b4c144a9b670786eb3d26466) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240502 08:20:00.138448 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(f1581261b4c144a9b670786eb3d26466): perf score=0.033356
I20240502 08:20:00.144946 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling FlushMRSOp(ad53a92fbcc041378bb1958c65e997db): perf score=0.033345
W20240502 08:20:00.171273 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:20:00.178082 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(f1581261b4c144a9b670786eb3d26466) complete. Timing: real 0.039s user 0.035s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5236,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1798,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":794,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":5408,"threads_started":2,"wal-append.queue_time_us":366}
I20240502 08:20:00.179165 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: FlushMRSOp(ad53a92fbcc041378bb1958c65e997db) complete. Timing: real 0.034s user 0.029s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1650,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1046,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":255,"threads_started":1,"wal-append.queue_time_us":353}
I20240502 08:20:00.180810 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling FlushMRSOp(581d2ed76e2e446eb84b94231ebb97f4): perf score=0.033353
I20240502 08:20:00.181854 12777 maintenance_manager.cc:419] P 29b035434087435cbaf4ebcac98eedee: Scheduling UndoDeltaBlockGCOp(ad53a92fbcc041378bb1958c65e997db): 4925 bytes on disk
I20240502 08:20:00.182332 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling FlushMRSOp(581d2ed76e2e446eb84b94231ebb97f4): perf score=0.033357
I20240502 08:20:00.182451 12667 maintenance_manager.cc:643] P 29b035434087435cbaf4ebcac98eedee: UndoDeltaBlockGCOp(ad53a92fbcc041378bb1958c65e997db) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
W20240502 08:20:00.198799 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:20:00.205377 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling FlushMRSOp(ad53a92fbcc041378bb1958c65e997db): perf score=0.033376
I20240502 08:20:00.242965 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: FlushMRSOp(581d2ed76e2e446eb84b94231ebb97f4) complete. Timing: real 0.061s user 0.032s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1230,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":840,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7596,"threads_started":1,"wal-append.queue_time_us":7362}
I20240502 08:20:00.245921 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(581d2ed76e2e446eb84b94231ebb97f4): 4840 bytes on disk
I20240502 08:20:00.250718 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(581d2ed76e2e446eb84b94231ebb97f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 08:20:00.253257 13038 maintenance_manager.cc:419] P 60b6e265e7754b7187307d05ef405dd4: Scheduling UndoDeltaBlockGCOp(f1581261b4c144a9b670786eb3d26466): 4586 bytes on disk
I20240502 08:20:00.254530 12929 maintenance_manager.cc:643] P 60b6e265e7754b7187307d05ef405dd4: UndoDeltaBlockGCOp(f1581261b4c144a9b670786eb3d26466) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240502 08:20:00.278430 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:20:00.294709 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: FlushMRSOp(ad53a92fbcc041378bb1958c65e997db) complete. Timing: real 0.089s user 0.045s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":18110,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":3873,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":874,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":29117,"threads_started":2,"wal-append.queue_time_us":6715}
I20240502 08:20:00.295960 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: FlushMRSOp(581d2ed76e2e446eb84b94231ebb97f4) complete. Timing: real 0.113s user 0.034s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3730,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":5350,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1168,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":20041,"threads_started":2,"wal-append.queue_time_us":9687}
I20240502 08:20:00.297209 13168 maintenance_manager.cc:419] P 6a0b05f3250d415590b0edf6b44fbe1f: Scheduling UndoDeltaBlockGCOp(ad53a92fbcc041378bb1958c65e997db): 4925 bytes on disk
I20240502 08:20:00.297974 13059 maintenance_manager.cc:643] P 6a0b05f3250d415590b0edf6b44fbe1f: UndoDeltaBlockGCOp(ad53a92fbcc041378bb1958c65e997db) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240502 08:20:00.298121 12908 maintenance_manager.cc:419] P e1dd14f7ca764a1490df419f13f2804a: Scheduling UndoDeltaBlockGCOp(581d2ed76e2e446eb84b94231ebb97f4): 4840 bytes on disk
I20240502 08:20:00.298851 12799 maintenance_manager.cc:643] P e1dd14f7ca764a1490df419f13f2804a: UndoDeltaBlockGCOp(581d2ed76e2e446eb84b94231ebb97f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240502 08:20:00.302543 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:00.407835 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:20:00.432521 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:20:00.445034 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:00.470292 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:00.483261 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:00.489988 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:20:00.507275 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:00.523430 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:00.562515 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:00.576193 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:00.577849 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:00.604882 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:20:00.620723 15419 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.1:33183, user_credentials={real_user=slave}} blocked reactor thread for 53264.2us
W20240502 08:20:00.668360 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:20:00.709462 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:00.712779 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:00.736989 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:00.786001 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:00.792814 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:00.855551 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:00.914512 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:00.945339 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:00.960628 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:00.975023 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:00.975683 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:00.984997 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:00.990736 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:00.997177 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:01.018682 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:01.028832 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:01.057339 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.073849 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:01.099308 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:01.121693 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.149566 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.155648 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.157418 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.179114 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.212319 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:01.222342 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.229255 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.235476 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.243927 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.249305 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.258491 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.259583 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.287743 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.308629 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.314119 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.381808 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.391247 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.409615 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:01.410729 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:01.421270 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:01.425204 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:01.434947 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:01.448974 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:01.452142 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:01.473335 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:01.483493 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:01.512677 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:01.550076 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.601317 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.605213 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:01.649001 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.717924 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.720602 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:01.727990 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:01.843369 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:01.856483 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:01.861040 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:01.876528 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:01.887562 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:01.890343 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:01.945667 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:02.026628 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:02.112998 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:02.132031 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:20:02.135684 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:02.349128 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:02.353101 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:02.490049 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:02.503549 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:02.542551 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:02.564538 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:02.607520 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:02.614624 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:02.618198 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:02.622210 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:02.628062 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:02.723984 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:02.741678 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:02.780553 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:02.846849 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:02.935170 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:02.980448 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:02.996399 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:03.003069 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:03.014719 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:03.026247 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:03.033929 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:03.044358 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:03.060700 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:20:03.079228 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:03.202309 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:03.213654 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:03.226096 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.233713 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.262076 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.330749 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.336360 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.337005 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:03.391934 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:03.392680 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:03.395546 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.400617 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.403987 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:03.424293 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:03.431591 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:03.457274 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:03.460052 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:03.463575 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:03.499848 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:03.512840 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.535715 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.564103 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.571684 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.584766 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.588275 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.683583 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.702045 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.715894 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.724575 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.775918 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:03.786751 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.827313 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.864008 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240502 08:20:03.865022 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.868901 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.872700 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:03.895162 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:03.896950 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:03.902477 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:20:03.910655 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:03.913475 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:03.920838 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:20:03.921412 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 08:20:03.922631 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.923172 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.932847 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.944648 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:03.962370 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:03.962925 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:03.966004 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:03.967294 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:03.971920 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 08:20:03.973165 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:03.980463 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:04.012454 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:20:04.032316 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:04.038640 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 08:20:04.039484 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:04.065346 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:04.212170 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:04.360591 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:04.384305 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:04.404556 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:04.438172 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:04.443809 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:04.493167 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:04.501384 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:04.533775 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:04.534330 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:04.538146 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:04.593480 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:04.610615 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:04.692013 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:20:04.768239 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:04.839066 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:04.947942 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:05.076468 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:05.103075 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:05.110708 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:05.129738 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:05.135201 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:05.143561 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:05.154553 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:05.161350 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:05.170343 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:05.223138 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:05.233040 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:05.320472 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:05.327665 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:05.520838 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:05.538100 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:05.549252 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:05.557134 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:05.570690 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:05.581727 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:05.582684 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:05.588158 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:05.589277 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:20:05.624385 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:05.644775 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:05.701745 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:05.735462 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:05.756058 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:05.763141 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:05.776049 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:05.832468 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:05.842392 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:05.850632 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:05.869839 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:05.915768 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:05.951490 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:05.954277 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:05.977989 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:05.979844 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:20:05.999064 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:06.002624 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:06.003386 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:06.021895 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:20:06.023124 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:06.030810 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:06.056268 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:06.074122 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:06.180523 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.204639 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.209384 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.251142 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.303809 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.324915 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.351953 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:06.370447 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.389082 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.417428 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.423082 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.434916 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.439808 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.456306 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.467641 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.469161 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:20:06.470674 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:06.473538 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:06.474148 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:20:06.475145 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:06.480758 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:06.502287 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.564545 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:20:06.569118 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:06.627012 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:06.642611 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.643393 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:20:06.657693 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:06.701215 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.770957 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.861059 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.862022 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:06.921557 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:20:06.929296 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:06.958993 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:06.966109 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:06.967003 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:07.009729 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:07.010126 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:07.024240 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:20:07.034348 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:07.052043 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:07.131726 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:07.326757 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:07.332753 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:07.478103 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:07.592716 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:07.603070 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:20:07.645174 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:07.652649 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:07.686025 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:07.686352 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:07.693475 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:07.734119 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:07.769070 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:07.770455 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:20:07.772564 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:07.800050 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:07.858664 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:07.869354 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:07.878029 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:07.952903 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:08.001617 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:20:08.093561 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:08.098716 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:08.110656 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:08.127605 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:20:08.132645 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:08.189129 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:08.210716 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:08.212028 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:08.219571 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.271870 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.273957 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:08.325680 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.328133 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:08.400637 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.425604 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.426091 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:08.449720 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.474788 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.500595 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:20:08.569123 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:08.597086 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:08.598376 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:08.610395 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:20:08.615953 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:08.623399 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:08.633867 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:08.663375 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:08.675468 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.686631 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.690577 12664 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.2:36933, user_credentials={real_user=slave}} blocked reactor thread for 55166.1us
W20240502 08:20:08.702066 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.758878 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.769158 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.788362 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.792409 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.801378 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.827487 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.836110 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.837939 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.841117 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.889732 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 08:20:08.915164 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.946820 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.949296 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:08.953985 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:20:09.006901 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:20:09.008150 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:09.020947 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:09.021342 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:20:09.022922 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:09.023852 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:09.023923 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:09.032799 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:20:09.044485 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:09.052961 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:09.059734 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 08:20:09.067656 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:09.068439 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:09.072177 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 08:20:09.072592 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:09.082911 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:09.106287 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 08:20:09.123250 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:09.131282 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:20:09.160280 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:09.195891 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:09.196107 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:09.256217 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:09.304497 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:09.423096 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:20:09.496965 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:09.515206 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:09.545428 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:09.553944 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:09.559687 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:09.577494 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:09.609310 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:09.622674 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:09.654996 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:20:09.859882 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:09.974524 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:09.978346 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:10.007401 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:10.049276 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:10.121863 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:10.173941 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:20:10.174681 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:10.187939 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:10.202270 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:10.205004 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:10.217996 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:10.227447 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:20:10.229588 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:10.234124 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:10.263671 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:10.307126 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:10.346987 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:10.428746 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:10.484403 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:10.614180 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:10.617419 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:10.621583 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:20:10.651050 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:10.682149 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:10.698714 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:10.699892 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:20:10.703510 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:10.738860 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:10.745159 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:10.758555 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:20:10.800336 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:10.847522 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:10.912243 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:10.924525 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:10.963725 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:10.974305 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:20:11.004230 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.033106 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.079593 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.093688 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:11.131953 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:20:11.141090 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:11.170128 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:11.175370 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.175944 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:20:11.186895 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:11.227447 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.234515 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:20:11.247593 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:11.250211 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:11.250483 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:11.256561 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:11.305048 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.308164 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.333962 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.352363 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.392953 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.413722 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.414201 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.422106 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.425943 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.435837 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.436436 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.447855 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.447978 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.455648 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.542519 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:11.567919 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.585773 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:20:11.644672 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:11.656637 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:11.681481 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:20:11.705852 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:11.711110 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:11.750888 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:11.766834 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:20:11.767863 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:11.770975 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.828588 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:11.831948 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:20:11.883927 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:11.980427 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:12.038066 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:12.184567 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:12.197525 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:12.201608 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:12.224439 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:12.247520 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:12.251487 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:20:12.265900 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:12.281350 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:12.355978 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:12.509136 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:12.588220 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:12.613804 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:12.637147 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:12.663552 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:12.674153 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:12.735391 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:12.782979 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:12.783762 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:12.834549 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:20:12.906903 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:12.909121 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:12.926715 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:12.929853 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:12.968912 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:13.008080 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:13.019034 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:13.030650 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:13.145212 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.185875 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.206758 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:13.227960 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:13.343329 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:20:13.357892 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:13.358750 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:13.369158 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:20:13.377084 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:13.377735 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:13.408949 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:13.426087 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:20:13.426200 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:13.459146 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:13.466912 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.479209 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.482300 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.494030 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.505007 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.585740 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:20:13.611728 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.657537 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:13.659386 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.667112 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.671825 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.699502 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.703274 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.729018 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.751219 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.759896 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.763806 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.771843 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.833142 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:13.839466 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:20:13.906617 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:13.925419 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 126 similar messages]
I20240502 08:20:13.944794 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:13.952656 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:13.972487 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:13.981168 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:14.000172 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:14.003151 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 08:20:14.027086 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:14.027851 12664 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.2:36933, user_credentials={real_user=slave}} blocked reactor thread for 60239.9us
W20240502 08:20:14.029263 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:20:14.032246 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:14.082896 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:20:14.084388 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:20:14.084950 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:14.105775 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:14.106482 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 08:20:14.107772 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:14.154721 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:14.156577 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:14.214025 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 08:20:14.230978 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:14.253433 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:14.264319 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:20:14.276408 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:14.348924 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:14.377106 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 08:20:14.383651 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:20:14.406282 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:14.406706 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:20:14.407232 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:14.432543 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:14.434677 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:14.468698 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:20:14.501945 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:14.502292 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:20:14.515287 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:14.557046 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:14.563292 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:14.616554 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:14.707695 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:14.815286 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:14.940203 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:20:14.941992 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:14.966850 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:15.000030 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:15.010007 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:15.015983 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:15.017867 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:15.054750 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:15.128163 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:15.154021 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:15.191195 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:15.249204 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:15.258368 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:15.271272 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:15.338536 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:15.416373 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:15.454895 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:15.542901 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:20:15.701248 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:15.705010 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:15.738288 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:20:15.745745 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:15.752492 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:15.781109 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:20:15.784600 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:15.796531 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:15.802956 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:15.845898 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:15.881150 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:20:15.890609 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:15.919170 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:15.919502 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:15.925925 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:15.926620 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:15.943810 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:15.981904 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.011195 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.058586 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.094822 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.101709 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.171747 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.181628 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.202566 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.204262 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:20:16.228456 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:16.230108 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:16.232434 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:16.234225 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:16.259157 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:16.270179 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.280347 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.289310 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.289445 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.341567 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.370488 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:20:16.376304 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:16.381727 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:16.385738 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:16.412565 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.477612 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.517408 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.575706 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.589962 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.606237 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.634977 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.663367 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.716240 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:20:16.758253 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:16.764631 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:16.767915 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:20:16.792418 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:16.799638 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:16.801671 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:16.846537 15688 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 56023.8us
W20240502 08:20:16.851153 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:16.922291 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:16.944290 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:16.974921 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:16.987529 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:17.033643 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:17.079952 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:20:17.090531 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:17.136508 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:17.225975 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:17.258713 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:20:17.271534 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:17.275089 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:17.275971 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:17.281519 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:17.314903 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:17.342850 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:17.361162 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:17.464210 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:17.524284 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:17.597182 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:17.684494 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:17.742839 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:20:17.747519 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:17.751627 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:17.761132 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:17.773018 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:17.776607 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:17.789264 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:17.796777 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:17.860817 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:20:17.871119 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:17.909087 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:17.923789 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:17.929809 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:17.944068 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:17.992288 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:18.102210 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.276857 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.277917 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.310925 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.365033 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:18.448971 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.509395 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.521832 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:18.522713 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.525364 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.535471 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:20:18.550390 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:18.586171 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:18.586294 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:20:18.595023 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:18.595448 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:18.615347 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:18.620757 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:20:18.633572 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:18.641824 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:18.669984 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.690654 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:20:18.710880 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:18.743908 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.746589 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.749306 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.750378 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.761366 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.783653 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.798487 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.813853 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.859985 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.906839 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.909065 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.917346 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:18.959703 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240502 08:20:18.979565 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:19.001034 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:19.006323 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:19.016960 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:20:19.023850 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:19.028281 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:19.035372 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:19.054783 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:20:19.080228 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:19.093667 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:20:19.096448 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:19.111605 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 08:20:19.123267 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:19.132867 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:19.148954 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:19.150190 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:19.174049 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:19.241971 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 08:20:19.243377 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:19.327639 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:20:19.356969 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:19.398650 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 08:20:19.400974 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:19.415625 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:19.448850 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:19.456156 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:19.466593 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:20:19.493428 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:19.496260 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:19.571188 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:19.581521 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:19.594066 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:19.619789 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:19.627115 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:19.738972 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:20:19.778082 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:19.898701 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:19.929896 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:19.930984 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:20:19.980182 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:19.980890 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:19.981149 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:19.987867 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:20.015743 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:20.018796 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:20.058984 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:20:20.062729 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:20.221956 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:20.251452 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:20.272671 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:20.297699 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:20.366398 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:20.396438 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:20:20.413341 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:20.426694 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:20.427245 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:20.451673 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:20:20.454926 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:20.476354 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:20.544440 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:20.553057 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:20.581845 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:20.609685 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:20.640292 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:20.649042 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:20.743647 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:20.750835 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:20.775723 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:20.808688 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:20.814432 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:20.816300 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:20.895637 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:20.971251 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:21.049527 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.061738 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:20:21.097510 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:21.107589 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:21.120703 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.135960 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:20:21.142867 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:21.148707 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:21.155849 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.160164 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.162917 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:20:21.187652 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:21.211285 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:21.222095 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.241333 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:20:21.260614 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:21.286720 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.292634 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.292982 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.314081 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:21.331370 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.332255 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:21.374300 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.387703 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.393795 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.394448 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.506057 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:20:21.540732 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:21.565212 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:21.587404 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:21.590655 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:21.592334 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:21.617760 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:21.619114 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:21.620294 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.641196 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.642741 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.655993 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.674204 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.676409 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.676515 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.676592 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:21.714111 12795 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.7:38851, user_credentials={real_user=slave}} blocked reactor thread for 52351.7us
W20240502 08:20:21.731951 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:21.808981 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:20:21.976166 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:21.977303 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:22.008839 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:22.009302 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:20:22.037429 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:22.062013 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:22.072916 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:22.087846 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:22.197113 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:22.249802 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:22.283316 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:22.323022 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:20:22.432915 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:22.452316 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:22.461500 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:22.487092 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:22.517833 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:22.542199 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:22.555814 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:22.558925 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:22.580327 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:22.581238 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:22.629112 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:22.663780 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:22.698364 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:22.713032 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:22.761862 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:22.850075 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:20:22.911635 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:22.934229 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:22.944525 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:22.956665 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:22.957809 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:22.975322 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:23.009789 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:23.036062 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:23.039316 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:23.061326 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:23.125164 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.202191 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.244387 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:23.251526 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.270197 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.344516 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.365521 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.383963 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.408556 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:23.436336 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:23.436597 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.441036 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.526160 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.531255 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.533721 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.555689 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.556206 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.567930 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.575099 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:23.584280 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:23.600006 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:23.600829 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:23.627388 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:23.642621 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:23.661171 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.661219 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:23.683840 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:23.713953 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:23.727061 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:23.792065 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.800252 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.802156 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.840972 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.885288 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.898124 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.947791 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:23.966609 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:23.996707 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 08:20:23.998289 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:24.030498 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:24.039328 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:24.043090 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:24.050361 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:24.051059 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:24.073777 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:20:24.080801 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:24.106379 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:24.129316 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:24.132735 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:20:24.136665 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:24.139338 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:24.149647 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:24.150288 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:24.151798 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:24.182091 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:20:24.187413 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:24.291252 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:24.339841 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:24.380180 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 08:20:24.397992 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:24.482880 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 08:20:24.534957 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:24.536237 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:24.557639 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:24.562705 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:24.573702 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:24.624122 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:24.626860 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:24.628098 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:24.653224 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 08:20:24.670841 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:24.777833 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:20:24.800877 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:24.866868 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:24.907066 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:24.967501 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:24.968554 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:24.995440 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:24.998700 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:25.003240 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:25.009200 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:25.016516 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:25.019706 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:25.160259 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:25.195875 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:25.274124 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:25.275559 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:25.412184 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:25.412915 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:25.429673 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:25.440302 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:25.447715 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:25.483695 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:25.489487 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:25.495031 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:25.499819 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:25.554320 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:25.557308 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:25.573843 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:25.613073 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:25.659389 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:25.717824 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:25.739238 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:25.792145 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:25.824815 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:25.848299 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:25.924571 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:25.936509 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:25.961402 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:25.980151 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:25.982236 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:25.989964 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:26.033766 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:26.066831 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:26.081172 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:26.086709 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:26.088300 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.097771 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:26.099207 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:26.108177 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:26.109457 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:26.128983 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.149231 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.154083 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.179304 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:26.184808 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:26.191733 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.197777 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:26.204011 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:26.241971 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.272886 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:20:26.329643 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.389333 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:26.393524 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.467514 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.502411 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:26.535244 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:26.554869 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.606456 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:26.608292 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:26.613615 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:26.636896 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:26.652676 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:26.652700 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:26.696919 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.733089 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:26.767555 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:26.770030 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.770819 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.786357 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.808245 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:26.854382 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:27.012324 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:27.052680 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:27.061414 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:27.073038 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:27.083695 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:20:27.096982 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:27.102809 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:27.124235 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:27.136409 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:27.149350 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:27.154651 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:27.190106 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:27.190340 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:27.403506 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:27.523022 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:27.546568 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:27.553210 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:27.564790 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:27.583060 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:27.606174 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:27.629992 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:27.641491 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:27.643563 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:27.645915 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:27.725557 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:27.739346 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:27.779001 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:27.874022 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:27.894730 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:27.955490 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:27.962507 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:27.984653 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:27.988575 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:28.041694 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:28.042577 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:28.050724 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:20:28.075467 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:28.084903 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:28.127571 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:28.130420 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:28.150233 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.177733 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.200260 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.249378 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.323242 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:28.351259 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.420495 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:28.429930 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.450166 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.500661 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.511371 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.541319 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.563162 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.615023 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.666646 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.682898 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:28.713526 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:28.729935 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:28.745252 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:28.752008 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.752120 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:28.758664 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:28.766708 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:28.778898 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:28.780305 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:28.784466 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.806919 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:28.816625 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:28.818002 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:28.837226 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.842602 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.908357 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:28.925622 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:28.966625 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:28.993324 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:29.005383 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:29.074695 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240502 08:20:29.095938 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:29.119876 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 08:20:29.123104 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:29.152873 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:29.180207 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:29.193464 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:29.201501 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:29.212421 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:29.222501 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:20:29.232674 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:29.242444 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:29.246322 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:20:29.250602 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:29.257124 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:29.264597 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:29.265771 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:29.297494 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:29.349296 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:29.463021 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 08:20:29.464241 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:20:29.489691 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:29.513839 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:29.554338 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:20:29.633409 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:29.663017 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:29.665263 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:29.674026 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 08:20:29.676071 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:29.705149 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:29.720198 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:29.720685 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:29.817481 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:29.843358 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:29.857700 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:29.870168 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:29.881484 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:29.948261 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:30.005056 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:30.072391 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:30.179566 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:30.202723 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:30.203306 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:30.219156 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:30.225267 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:30.247149 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:30.281571 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:30.286255 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:30.296200 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:30.365568 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:30.458153 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:30.478350 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:30.498138 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:30.519079 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:30.557842 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:30.637701 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:30.648530 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:30.656934 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:30.662964 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:30.664261 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:30.680450 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:30.684791 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:30.695286 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:30.710819 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:30.715291 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:30.738274 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:30.740809 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:30.760178 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:30.770530 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:30.781883 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:30.815701 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:30.829106 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:30.842370 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:30.940693 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:30.952721 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:30.968020 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:31.003906 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.197780 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.212503 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.218935 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.244310 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.280059 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:31.323096 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:31.340677 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:31.365090 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:31.370942 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:31.374241 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:31.382730 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:31.426141 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:31.427376 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:31.431643 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:31.449466 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:31.456185 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.456564 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.456729 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:31.464093 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:31.472476 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:31.472549 15419 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.2:36933, user_credentials={real_user=slave}} blocked reactor thread for 80764us
W20240502 08:20:31.480939 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.535537 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.554020 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.577806 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.670217 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.703814 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:20:31.733676 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.747385 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:31.826892 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:31.895825 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:31.897275 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:31.905565 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:31.913694 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:20:31.921859 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:31.933333 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:31.934609 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:31.951593 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:31.969419 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:31.969502 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:31.976310 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:32.067590 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:32.097807 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:32.173262 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:32.318251 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:32.321067 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:32.325760 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:32.330981 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:32.331386 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:32.360146 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:32.404806 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:32.432176 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:32.440783 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:32.443280 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:32.481447 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:32.572042 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:32.596412 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:32.661121 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:32.700587 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:32.741644 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:32.795352 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:32.857645 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:20:32.872534 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:32.906549 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:32.909162 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:32.910704 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:32.921288 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:32.922482 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:20:32.927526 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:32.950690 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:33.003618 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:33.018184 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:33.046542 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.137533 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.228953 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.241139 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:20:33.327584 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:33.337926 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:33.339243 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:33.369851 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:33.371757 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:33.389891 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:33.390547 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:20:33.396467 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:33.406116 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.423007 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.430191 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:20:33.431787 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:33.452733 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:33.464205 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:33.470202 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.585813 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.591217 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.713387 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.725011 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.751158 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.799932 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.805641 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.874794 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.922600 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.929675 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:33.965875 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:33.967275 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:33.976853 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:20:34.035415 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:34.057749 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:34.076004 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:20:34.079826 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:34.084561 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:20:34.097577 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:34.114831 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:20:34.125447 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:20:34.136700 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:34.138208 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:34.145334 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 130 similar messages]
I20240502 08:20:34.152549 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:34.156967 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:34.174160 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:34.189157 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:20:34.195176 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:34.207991 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:34.262394 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:34.287693 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:20:34.298547 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 08:20:34.322588 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:34.365749 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:34.369258 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:20:34.381775 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:34.399560 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:34.463680 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:34.533648 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:34.541790 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 08:20:34.558748 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:34.559867 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:34.569680 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:34.583112 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:20:34.587981 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:34.594730 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:34.631402 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:34.642962 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:34.667363 15819 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 48225.4us
W20240502 08:20:34.684558 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:34.694968 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:20:34.701725 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:34.774154 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 08:20:34.774281 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:34.776281 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:34.928092 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:34.943717 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:20:34.998534 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:35.000949 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:35.009464 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:35.062530 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:20:35.069721 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:35.078624 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:35.089068 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:20:35.091725 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:35.098107 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:20:35.102677 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:35.177568 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:35.184939 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:35.196564 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:35.224488 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:35.276536 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:35.311450 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:35.428447 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:35.444389 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:35.478768 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:35.484907 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:35.500304 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:35.522644 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:20:35.549137 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:35.583415 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:35.586694 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:35.740417 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:35.741051 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:35.872578 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:35.873899 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:35.885110 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:35.891991 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:35.904174 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:35.917682 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:20:35.942718 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:35.949709 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:20:35.954798 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:35.958390 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:35.968914 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:35.984194 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:35.986898 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:36.010741 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:20:36.015388 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:36.020614 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:36.085289 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.164500 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.235821 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.262112 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.277565 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:36.279138 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.299544 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.350345 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.361707 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.437908 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.476330 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.525096 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.525725 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.550899 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:36.667521 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:36.673362 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:36.676218 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:36.682945 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:36.687804 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:36.716167 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.726612 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.729243 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:20:36.729976 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:36.745234 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:20:36.757310 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:36.759594 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:36.782438 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.788182 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:20:36.824126 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.836297 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.841132 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:36.846192 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:37.044545 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:37.048228 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:37.074324 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:37.081660 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:20:37.112392 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:37.114064 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:37.115029 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:37.121590 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:37.128504 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:37.161796 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:37.168023 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:37.206214 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:37.243031 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:37.243155 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:37.333438 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:37.401405 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:37.418735 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:37.443620 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:37.559957 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:37.570772 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:37.592960 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:37.607259 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:37.621218 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:37.622985 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:20:37.632230 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:37.701133 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:37.702900 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:37.719905 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:37.741330 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:20:37.745164 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:37.812608 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling FlushMRSOp(f031f5fedb904eaf83a4902092280295): perf score=0.033338
I20240502 08:20:37.909642 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: FlushMRSOp(f031f5fedb904eaf83a4902092280295) complete. Timing: real 0.096s user 0.064s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":1019,"dirs.queue_time_us":296,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1476,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":818,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2214,"threads_started":3,"wal-append.queue_time_us":316}
I20240502 08:20:37.912952 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling UndoDeltaBlockGCOp(f031f5fedb904eaf83a4902092280295): 4925 bytes on disk
I20240502 08:20:37.913060 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling FlushMRSOp(bf504f2522a147c198f99debdfe03d1e): perf score=0.033380
I20240502 08:20:37.918234 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: UndoDeltaBlockGCOp(f031f5fedb904eaf83a4902092280295) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240502 08:20:37.944733 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:37.951218 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:37.957791 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:37.974401 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:37.978159 12795 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.4:32979, user_credentials={real_user=slave}} blocked reactor thread for 52101.2us
W20240502 08:20:38.038658 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:38.040364 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: FlushMRSOp(bf504f2522a147c198f99debdfe03d1e) complete. Timing: real 0.127s user 0.078s sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":8061,"dirs.queue_time_us":6372,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":6616,"drs_written":1,"lbm_read_time_us":158,"lbm_reads_lt_1ms":4,"lbm_write_time_us":899,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":27720,"threads_started":3,"wal-append.queue_time_us":12916}
I20240502 08:20:38.043627 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling UndoDeltaBlockGCOp(bf504f2522a147c198f99debdfe03d1e): 4840 bytes on disk
I20240502 08:20:38.044780 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: UndoDeltaBlockGCOp(bf504f2522a147c198f99debdfe03d1e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240502 08:20:38.194551 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.384584 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.390421 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.393589 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.454310 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:38.463068 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.467178 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.501081 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.506691 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.512301 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.539028 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.563292 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:38.594897 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.595417 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:38.606034 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:20:38.627022 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:38.627655 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:38.649161 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:20:38.664395 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:38.674733 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:38.707291 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:38.716287 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.743176 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:20:38.745024 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:38.752967 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:38.805168 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling FlushMRSOp(db051ed0438742319f741a565834b644): perf score=0.033394
W20240502 08:20:38.812184 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:20:38.900076 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: FlushMRSOp(db051ed0438742319f741a565834b644) complete. Timing: real 0.095s user 0.032s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":313,"dirs.run_cpu_time_us":299,"dirs.run_wall_time_us":11424,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":756,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":475,"threads_started":2,"wal-append.queue_time_us":437}
I20240502 08:20:38.903095 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling UndoDeltaBlockGCOp(db051ed0438742319f741a565834b644): 4586 bytes on disk
I20240502 08:20:38.903743 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: UndoDeltaBlockGCOp(db051ed0438742319f741a565834b644) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
W20240502 08:20:38.910148 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.936960 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:38.975227 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.036401 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.099191 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.114993 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:39.118633 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.119657 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.122982 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:20:39.124028 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:39.142189 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.147585 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:20:39.163206 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:39.164093 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling FlushMRSOp(91f185fbf27049a7992daa1d905e5575): perf score=0.033374
I20240502 08:20:39.177467 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:39.188796 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:39.189508 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 129 similar messages]
I20240502 08:20:39.209154 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:39.223507 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:20:39.226389 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:39.249553 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:39.261030 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:20:39.284943 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:39.290685 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: FlushMRSOp(91f185fbf27049a7992daa1d905e5575) complete. Timing: real 0.126s user 0.034s sys 0.050s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1312,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1528,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":68111,"threads_started":1,"wal-append.queue_time_us":68201}
I20240502 08:20:39.299644 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling UndoDeltaBlockGCOp(91f185fbf27049a7992daa1d905e5575): 4840 bytes on disk
W20240502 08:20:39.300581 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:20:39.301787 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: UndoDeltaBlockGCOp(91f185fbf27049a7992daa1d905e5575) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240502 08:20:39.316669 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.330406 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.355363 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:20:39.431915 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.463116 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:20:39.512465 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.535794 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.544147 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.569674 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.584161 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:20:39.596410 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 08:20:39.641366 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:39.723749 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:39.760699 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:39.777046 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:39.778658 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 08:20:39.845183 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:39.863593 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:39.888135 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:39.903729 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:20:39.912169 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:39.942692 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:39.945240 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:39.962879 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:20:39.970628 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:40.050103 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:40.055630 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:40.094893 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:40.105067 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:40.232244 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:40.237085 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:40.254117 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:40.282922 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:20:40.296629 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:40.302228 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:40.307153 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:40.354882 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:40.376590 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:40.390929 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:20:40.414149 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:40.435344 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:40.606778 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:40.674216 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:40.675558 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:40.685901 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:40.770615 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:20:40.817863 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:40.824751 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:20:40.852174 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:40.873736 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:40.891450 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:20:40.894171 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:40.895756 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:20:40.905592 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:40.908087 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:40.909370 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:40.957455 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:40.983258 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.082530 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.089921 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.155808 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.195642 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.196209 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.203128 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:41.224215 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:41.229036 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:41.234848 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:20:41.237890 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:41.254897 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:20:41.275414 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:41.275535 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:41.300776 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:41.315212 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:20:41.334025 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:41.339615 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.390828 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.411813 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.455999 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.502673 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.530915 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.539732 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.590354 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:41.591182 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.594303 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:41.646018 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.659251 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:41.720258 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:20:41.788518 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:41.842654 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:41.843593 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:41.878016 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:20:41.889482 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:41.895861 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:41.915378 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:41.944269 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.948201 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:20:41.950538 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:41.964637 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:41.973528 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:42.008175 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:42.062829 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:42.084659 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:20:42.103760 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:42.123770 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:42.136917 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:42.140202 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:42.189888 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:42.322840 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:42.358673 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:42.388296 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:42.430837 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:42.443750 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:42.444023 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:42.490430 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:20:42.528157 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:42.530956 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:42.554019 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:42.575338 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:42.575726 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:42.577183 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:42.608423 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:42.630692 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:42.635582 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:42.710654 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:42.818382 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:42.835501 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:42.966018 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:43.000926 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:43.015300 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:43.022202 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:43.042364 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:43.042516 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:43.081481 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:43.082095 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:43.138684 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.212169 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:43.261929 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.285984 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:20:43.408684 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:43.436319 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:43.444206 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:43.451872 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.457064 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:43.470261 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:20:43.488086 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:43.488436 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:43.527230 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:20:43.547745 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:43.573669 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:20:43.595170 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:43.616868 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.623651 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.663038 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.729272 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.732445 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:43.746143 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.767820 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.800274 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.809559 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.865620 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.878690 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.879207 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:43.911724 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:43.918949 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:20:43.930996 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:43.942946 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:20:43.961493 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:43.961894 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:43.975451 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:43.996551 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:43.999380 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:44.005782 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:20:44.041292 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:44.053865 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.136901 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.197774 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.254228 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 08:20:44.255287 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:20:44.256116 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:44.267706 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.285210 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:44.316148 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:44.354446 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.368800 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 08:20:44.373757 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:44.402905 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:44.403805 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:20:44.405925 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:44.406803 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:44.426868 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:20:44.431495 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:44.435289 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:44.445163 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.511942 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.514901 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.520655 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:20:44.545377 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:20:44.560189 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:44.589973 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.612423 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.613334 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 08:20:44.684417 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.687242 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:20:44.715008 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.736436 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.763614 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:44.831331 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 08:20:44.848675 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:44.872987 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:44.917671 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:45.013687 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:45.114795 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:20:45.145740 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:45.159957 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:45.171598 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:45.185932 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:45.187810 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:45.194977 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:20:45.195751 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:45.242569 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:45.248394 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:45.253610 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:45.276916 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:20:45.282258 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:45.298293 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:45.415452 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:45.560407 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:20:45.631891 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:45.660967 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:45.664835 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:45.671319 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:45.702327 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:45.726205 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:20:45.743139 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:45.746681 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:45.748636 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:20:45.753072 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:45.917137 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:45.927351 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:45.934494 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:45.934779 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:45.956686 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.003042 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.052796 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.061147 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:20:46.094874 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:46.099905 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:46.102084 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.108537 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:46.119614 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:46.156417 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:46.160010 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:46.174445 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:46.174484 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:46.209622 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.228940 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.283105 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.335745 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.351009 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:46.358681 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.407097 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.428479 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.469962 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.482302 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.498152 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.535007 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.547235 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.582691 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:46.585531 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:20:46.596011 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:46.610450 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:46.640163 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:46.644071 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:46.657886 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:46.683219 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:46.728968 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:46.766403 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.797636 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:46.948141 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.977648 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:46.980335 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:47.000782 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:47.007251 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:47.043545 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:20:47.071595 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:47.071786 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:47.075140 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:47.090292 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:47.090400 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:47.100771 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:47.112670 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:20:47.117858 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:47.119099 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:47.140282 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:47.156359 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:47.177219 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:47.190838 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:20:47.199039 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:47.307780 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:47.467511 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:47.497881 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:47.499351 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:47.521342 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:20:47.528913 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:47.619093 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:47.675374 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:47.686208 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:47.771431 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:47.802901 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:47.814579 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:47.814710 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:47.849539 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:47.849931 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:47.864406 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:47.893981 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:47.936296 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:47.970536 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:48.043071 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:48.078047 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:48.153473 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.211927 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:48.304760 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:48.329463 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:48.329501 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:48.342489 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:48.345651 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:48.398661 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:48.431128 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:20:48.438360 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:48.448778 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.468868 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:48.473191 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:48.548146 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:48.602006 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.602861 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.692839 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.697417 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.718643 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.721571 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.725243 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.784004 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.804991 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.805552 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:48.806375 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.841235 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.842677 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.850167 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:48.856623 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:48.880970 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:48.893436 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:48.916247 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:48.917958 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:48.947328 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:48.952863 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:48.957437 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:48.967437 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:48.997597 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:49.007823 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:49.039952 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.087685 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:49.184448 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:20:49.222738 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.250396 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.271487 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 08:20:49.272496 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.289383 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:20:49.341547 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:49.348862 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:49.370343 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:49.377084 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.377173 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 08:20:49.382861 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:49.396342 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:49.408825 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.432878 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:20:49.434831 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:49.461401 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:49.473762 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.489255 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:20:49.514379 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.516288 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.551151 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.557559 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.561347 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.656248 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:49.665469 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.682760 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.735929 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 08:20:49.815912 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:20:49.850615 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 08:20:49.865741 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:49.891119 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:49.953678 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:49.997123 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:20:49.997386 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:49.999372 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:50.032167 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:50.036651 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:50.037067 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:50.069890 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:50.115885 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:50.120956 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:20:50.125769 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:50.148003 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:50.163146 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:50.204147 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:50.283651 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:50.347931 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:50.520627 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:50.529752 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:50.607904 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:20:50.659463 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:50.700117 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:50.718015 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:50.723824 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:50.740767 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:50.759979 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:50.774724 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:50.797281 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:50.804837 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:50.811677 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:50.891535 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:50.962566 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.018595 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.056439 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.071847 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.087402 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.105566 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.113447 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:51.121096 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:20:51.124985 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:51.140398 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:51.140779 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:51.140923 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:51.143798 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:20:51.178781 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:51.206048 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:51.209664 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:20:51.220314 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:51.221598 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.229220 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.242215 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.242606 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.242710 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:51.268425 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.331184 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.344846 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.380213 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.427675 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.585209 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:51.586977 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:51.594383 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:51.616286 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:51.638588 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:51.640147 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:51.659372 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:20:51.670691 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:51.695256 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:51.738950 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.755988 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.785817 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.790057 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:51.797376 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:20:51.810094 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.810122 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.903935 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.931736 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:51.976171 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:52.010838 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:20:52.013314 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:52.022297 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:52.035648 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:52.036980 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:52.077695 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:52.078630 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:20:52.093241 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:52.095758 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:52.129827 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:20:52.137158 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:52.173856 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:52.291430 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:52.348242 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:52.371876 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:52.380581 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:52.414712 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:52.454378 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:20:52.461663 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:52.476372 15801 maintenance_manager.cc:419] P 9333cb10227b4801b4fa7c24e22e131b: Scheduling FlushMRSOp(f1581261b4c144a9b670786eb3d26466): perf score=0.033402
I20240502 08:20:52.494634 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:52.503002 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:52.548470 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:52.575162 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:52.578158 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:52.603742 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:20:52.646507 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:52.651883 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:52.667336 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:20:52.683086 15691 maintenance_manager.cc:643] P 9333cb10227b4801b4fa7c24e22e131b: FlushMRSOp(f1581261b4c144a9b670786eb3d26466) complete. Timing: real 0.206s user 0.055s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":26771,"dirs.queue_time_us":9354,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":11748,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3559,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":45455,"threads_started":3,"wal-append.queue_time_us":9465}
I20240502 08:20:52.685398 15801 maintenance_manager.cc:419] P 9333cb10227b4801b4fa7c24e22e131b: Scheduling FlushMRSOp(b87724cd8beb4449b272f972c85a6666): perf score=0.033335
I20240502 08:20:52.759708 15691 maintenance_manager.cc:643] P 9333cb10227b4801b4fa7c24e22e131b: FlushMRSOp(b87724cd8beb4449b272f972c85a6666) complete. Timing: real 0.073s user 0.060s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1505,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":804,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1075,"threads_started":1,"wal-append.queue_time_us":1012}
I20240502 08:20:52.762383 15801 maintenance_manager.cc:419] P 9333cb10227b4801b4fa7c24e22e131b: Scheduling UndoDeltaBlockGCOp(b87724cd8beb4449b272f972c85a6666): 4586 bytes on disk
I20240502 08:20:52.763129 15691 maintenance_manager.cc:643] P 9333cb10227b4801b4fa7c24e22e131b: UndoDeltaBlockGCOp(b87724cd8beb4449b272f972c85a6666) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240502 08:20:52.765262 15801 maintenance_manager.cc:419] P 9333cb10227b4801b4fa7c24e22e131b: Scheduling UndoDeltaBlockGCOp(f1581261b4c144a9b670786eb3d26466): 4586 bytes on disk
I20240502 08:20:52.766860 15691 maintenance_manager.cc:643] P 9333cb10227b4801b4fa7c24e22e131b: UndoDeltaBlockGCOp(f1581261b4c144a9b670786eb3d26466) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240502 08:20:52.787968 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:52.825343 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling FlushMRSOp(0fa42a21c83942f789dcebed741b52ae): perf score=0.033384
W20240502 08:20:52.825613 13056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.1:33183, user_credentials={real_user=slave}} blocked reactor thread for 55362.6us
W20240502 08:20:52.832159 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:52.842375 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:52.882440 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: FlushMRSOp(0fa42a21c83942f789dcebed741b52ae) complete. Timing: real 0.057s user 0.036s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":5440,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1377,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":873,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6056,"threads_started":2,"wal-append.queue_time_us":793}
I20240502 08:20:52.885172 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling UndoDeltaBlockGCOp(0fa42a21c83942f789dcebed741b52ae): 4925 bytes on disk
I20240502 08:20:52.885717 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: UndoDeltaBlockGCOp(0fa42a21c83942f789dcebed741b52ae) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240502 08:20:52.888556 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling FlushMRSOp(a3583c839d364bae819928eca86010ec): perf score=0.033385
W20240502 08:20:52.891479 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:52.912775 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:52.946234 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: FlushMRSOp(a3583c839d364bae819928eca86010ec) complete. Timing: real 0.057s user 0.041s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":3153,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":235,"threads_started":1,"wal-append.queue_time_us":320}
I20240502 08:20:52.948838 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling UndoDeltaBlockGCOp(a3583c839d364bae819928eca86010ec): 4586 bytes on disk
I20240502 08:20:52.949406 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: UndoDeltaBlockGCOp(a3583c839d364bae819928eca86010ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240502 08:20:52.988538 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:53.020673 15801 maintenance_manager.cc:419] P 9333cb10227b4801b4fa7c24e22e131b: Scheduling FlushMRSOp(5be94bf194b04e1d965274a968f30fbf): perf score=0.033347
W20240502 08:20:53.035928 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:53.132766 15691 maintenance_manager.cc:643] P 9333cb10227b4801b4fa7c24e22e131b: FlushMRSOp(5be94bf194b04e1d965274a968f30fbf) complete. Timing: real 0.112s user 0.065s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1170,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":906,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6390,"threads_started":1,"wal-append.queue_time_us":6295}
I20240502 08:20:53.135437 15801 maintenance_manager.cc:419] P 9333cb10227b4801b4fa7c24e22e131b: Scheduling UndoDeltaBlockGCOp(5be94bf194b04e1d965274a968f30fbf): 4586 bytes on disk
I20240502 08:20:53.136103 15691 maintenance_manager.cc:643] P 9333cb10227b4801b4fa7c24e22e131b: UndoDeltaBlockGCOp(5be94bf194b04e1d965274a968f30fbf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240502 08:20:53.214484 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:53.298033 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:53.314570 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:53.345609 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.374202 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.401310 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:53.409401 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:53.424543 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:53.432865 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:53.438040 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:53.443497 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.471350 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.474557 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:53.481981 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:53.504124 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:53.519471 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:53.556568 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:53.583650 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:53.618430 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.700143 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.743698 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.812980 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.819465 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:53.833649 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.843043 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.862916 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.871251 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.871320 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:53.939497 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:53.949867 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:53.980175 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:53.999313 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:54.013803 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:54.018266 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:54.029273 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:54.034085 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:54.049655 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:54.058279 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:54.071012 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:54.148573 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.201272 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.201835 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:20:54.251214 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.269109 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.272338 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240502 08:20:54.281128 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.339252 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:20:54.349529 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.379806 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240502 08:20:54.381809 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.393750 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:54.412039 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:54.432363 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:54.437883 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:54.458969 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:54.492379 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:54.495278 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.503747 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:20:54.542927 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:20:54.545478 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:54.576133 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:54.590443 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.597417 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.608075 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.627580 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.667762 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.709623 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.798885 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.814139 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 08:20:54.824272 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:20:54.828738 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:54.900543 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 08:20:54.902261 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:54.923838 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:54.925122 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:54.929323 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:54.935020 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:54.947328 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:54.969969 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:54.970254 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:55.038069 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:55.046315 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:55.062628 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:55.141252 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:55.274044 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:20:55.362318 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:55.362677 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:55.394843 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:55.395195 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:55.404228 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:55.411055 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:55.416921 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:55.434788 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:55.437052 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:55.449275 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:55.489535 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:55.494043 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:55.501047 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:55.549943 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:55.650579 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:55.682744 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:55.784222 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:55.850746 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.031518 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.061558 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:56.090597 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:20:56.091770 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.105055 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.108749 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:56.119009 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:56.124567 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:56.135777 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.145323 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:56.155632 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:56.164837 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.173930 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:56.181529 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:56.200639 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:56.201876 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:56.248618 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:56.260010 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.291201 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:56.302747 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:56.326400 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.366081 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.417342 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.435750 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.449910 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.467247 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.477874 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.537191 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.578585 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.605190 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:56.617842 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.647397 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.649390 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.652508 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:56.678836 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:56.711997 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:56.728780 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:56.730202 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:56.742748 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:56.766690 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:56.779135 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:56.790498 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:56.911828 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:56.925804 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:20:56.979296 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:57.061247 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:57.103744 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:57.111768 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:57.118600 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:57.166859 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:57.178412 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:57.192754 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:57.199191 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:57.201172 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:57.211184 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:57.243391 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:57.247337 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:57.252992 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:57.258919 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:57.258936 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:57.279388 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:57.447098 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:57.485908 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:57.509939 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:57.521272 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:57.602684 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:57.631261 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:20:57.662786 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:57.682287 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:57.682859 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:57.691452 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:57.701431 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:57.730016 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:57.762677 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:57.812242 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:57.823777 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:57.834997 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:57.907989 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:57.995693 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:58.010988 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.063032 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:58.084246 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:58.102182 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:58.112335 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:58.130512 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:58.143332 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:58.144340 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:58.148280 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.159132 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:58.167068 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:58.168565 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:58.202070 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:58.311260 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.322487 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.414479 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.426281 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.445353 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.488816 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:58.550621 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.622931 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.644083 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.696883 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:58.721933 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:58.757694 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:58.793093 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:20:58.800420 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:58.803503 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.810562 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:58.826843 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:58.838184 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:58.874912 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:58.880147 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:58.885109 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:58.890275 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.931885 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.942126 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.947822 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.959867 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:58.992843 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:59.000854 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.149751 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.157421 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:59.197510 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:59.217975 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:20:59.229847 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:59.245065 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:59.267482 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.277060 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:59.277626 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:20:59.282423 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:59.297080 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:59.307193 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:20:59.317445 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240502 08:20:59.318737 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.407341 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.418956 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:20:59.433197 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:20:59.433976 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:59.459450 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.509958 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:20:59.527873 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.533846 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.544458 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.589244 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.686784 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:59.724643 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:20:59.762607 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:59.775302 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:59.787297 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:20:59.789510 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:20:59.820472 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:20:59.825887 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:20:59.835928 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:20:59.840811 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.861882 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.869091 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 08:20:59.880491 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:59.886304 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:20:59.897763 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:20:59.910553 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:20:59.913586 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:20:59.914410 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:20:59.914772 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:20:59.948185 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240502 08:20:59.974810 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:00.175526 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:00.232239 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:21:00.249809 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:00.255508 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:00.291860 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:00.292963 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:00.302486 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:00.306516 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:00.306520 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:00.365849 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:00.367998 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:21:00.392328 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:00.438251 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:00.556982 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:21:00.642786 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:00.647717 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:00.668814 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:00.740324 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:00.798522 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:21:00.818076 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:00.823287 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:00.843538 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:21:00.854794 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:00.861402 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:00.885673 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:21:00.887809 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:00.898454 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:21:00.903594 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:00.911733 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:00.915285 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:21:00.919428 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:00.936573 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:00.968312 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:21:01.073756 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.087909 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.088124 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.142292 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.161186 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.192371 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.202508 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:01.379837 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.394490 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.445637 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.484063 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.484767 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.535161 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:21:01.550141 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:01.581624 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:01.586648 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:01.598431 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:01.627254 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:01.628383 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.638407 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.654419 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:21:01.658581 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:01.674712 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.680291 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.694269 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:01.721592 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:01.727141 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:21:01.854565 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.874127 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.881494 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.888183 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.955125 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.957281 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:01.975390 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:21:02.024648 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:02.033234 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:21:02.048986 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:02.056819 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:02.061615 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:02.090888 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:02.111310 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:02.114547 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:02.135059 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:02.164002 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:02.215181 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:02.330336 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:02.369694 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:02.407790 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:21:02.441130 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:02.448659 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:21:02.481331 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:02.486956 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:02.495712 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:02.510782 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:02.542644 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:21:02.559100 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:02.560406 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:02.604701 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:02.615658 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:02.655223 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:02.686633 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:02.756732 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:02.799952 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:02.881199 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:02.960569 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:21:02.985158 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:02.991191 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:03.011518 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:03.019585 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:03.020808 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:03.034272 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:03.050683 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:03.065694 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:03.137269 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.198218 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.321866 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:03.402050 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:03.440048 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:03.448532 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:03.451598 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:03.467227 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.467986 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:03.478340 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:03.491701 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:03.501437 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:03.502326 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:03.542213 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:03.556418 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:03.556517 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.572762 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.582145 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.625247 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:03.626111 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:03.634912 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.646203 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.664953 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.810818 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:03.825500 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.835070 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.838126 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.868387 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.875981 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.930931 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.931250 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:03.932137 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.022212 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:04.054563 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:04.057410 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.060079 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:04.079509 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:04.105288 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:04.123064 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:04.136299 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:04.162720 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.184108 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.195255 12925 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.7:38851, user_credentials={real_user=slave}} blocked reactor thread for 81369.8us
W20240502 08:21:04.210410 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:21:04.217074 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:04.233777 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:04.303540 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.344173 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.361992 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240502 08:21:04.363282 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.365399 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.369712 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:04.457140 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:21:04.459591 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.491384 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:21:04.517251 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240502 08:21:04.535535 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:04.587038 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:04.590322 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:04.613508 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:04.616014 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:04.634864 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:04.636102 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:04.647787 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:04.654450 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:04.713657 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.718143 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.750571 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.781996 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:04.811506 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.873270 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:21:04.911589 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240502 08:21:04.965585 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:04.968417 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 08:21:04.969717 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:05.078680 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:05.087514 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:21:05.088678 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:05.092906 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:05.094753 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:05.103546 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:05.130450 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:21:05.132836 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:05.147459 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:05.165800 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:05.184113 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:21:05.190971 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:05.248111 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:05.279979 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:05.285544 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:05.304545 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:05.306385 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:05.432037 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:05.623005 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:05.627239 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:05.628820 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:05.640071 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:21:05.651115 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:05.659070 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:05.690034 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:05.690377 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:05.747042 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:05.798612 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:05.803062 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:05.872861 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:05.939287 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.005061 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:21:06.037567 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:06.044721 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:06.055624 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:21:06.065480 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:06.071067 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:06.073582 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:06.104502 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:06.105486 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.112583 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:21:06.130932 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:06.141585 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:06.175212 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.203902 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.212842 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:06.217535 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:06.218994 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.241957 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.313588 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.315488 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:06.330248 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.339694 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.345432 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.383194 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.415803 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.452276 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.485121 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.555863 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.626310 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.641381 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.722884 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:06.725056 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.739955 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.769407 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:21:06.807477 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:06.836741 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:21:06.838816 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:06.844619 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:06.852257 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:06.866281 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:06.873675 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:06.909994 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:06.938917 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.941466 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:06.944831 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:07.045670 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:07.155045 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:07.244253 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:07.252230 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:21:07.257570 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:07.258731 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:07.265352 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:21:07.297755 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:07.325650 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:07.333278 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:07.342911 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:07.348623 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:21:07.354132 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:07.360369 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:07.405300 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:07.465026 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:07.471067 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:07.490346 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:21:07.516825 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:07.518908 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:07.591148 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:07.607774 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:07.663425 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:07.697649 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:07.712693 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:07.734433 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:07.737151 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:07.739128 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:21:07.744400 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:07.748829 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:21:07.794927 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:07.826928 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:07.859831 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:07.914528 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:08.021065 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:21:08.212322 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:08.221949 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:08.223129 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:08.241930 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:08.250988 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:08.258510 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:08.260253 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:08.318869 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:08.352762 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.396365 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.486873 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:08.504110 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.550092 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.550217 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.560722 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.579402 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.585672 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.588423 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.594255 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.598259 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.619174 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:08.657963 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:08.704764 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.749217 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:21:08.749828 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:08.777976 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:21:08.781458 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:08.794730 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.794876 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:21:08.795768 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:08.811971 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:08.823290 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:08.848234 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:08.878306 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:08.884562 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:08.901329 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:08.992830 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.041016 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.085949 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.088454 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.156764 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.256865 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.268261 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:21:09.302114 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.318398 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:09.323210 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.352674 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:21:09.357529 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling FlushMRSOp(660b389a179f49eeb3007dea6ce078e9): perf score=0.033373
I20240502 08:21:09.377208 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling FlushMRSOp(1f07b2caf1754c469a8e33382f615019): perf score=0.033387
W20240502 08:21:09.393359 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240502 08:21:09.401671 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.404940 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:21:09.456626 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:09.477909 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: FlushMRSOp(660b389a179f49eeb3007dea6ce078e9) complete. Timing: real 0.119s user 0.057s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":21486,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":2696,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":769,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":35968,"threads_started":2,"wal-append.queue_time_us":11033}
I20240502 08:21:09.481449 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling UndoDeltaBlockGCOp(660b389a179f49eeb3007dea6ce078e9): 4586 bytes on disk
I20240502 08:21:09.482111 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: UndoDeltaBlockGCOp(660b389a179f49eeb3007dea6ce078e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 08:21:09.484270 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling FlushMRSOp(581d2ed76e2e446eb84b94231ebb97f4): perf score=0.033353
I20240502 08:21:09.491001 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:09.503849 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:21:09.507635 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.515802 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240502 08:21:09.564739 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:09.583773 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:09.596168 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:09.605619 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:21:09.609200 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: FlushMRSOp(581d2ed76e2e446eb84b94231ebb97f4) complete. Timing: real 0.119s user 0.039s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1561,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":3014,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":838,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10327,"threads_started":1,"wal-append.queue_time_us":10219}
I20240502 08:21:09.612141 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling UndoDeltaBlockGCOp(581d2ed76e2e446eb84b94231ebb97f4): 4840 bytes on disk
I20240502 08:21:09.612815 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: UndoDeltaBlockGCOp(581d2ed76e2e446eb84b94231ebb97f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 08:21:09.665632 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:09.672518 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:09.680559 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:21:09.680982 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: FlushMRSOp(1f07b2caf1754c469a8e33382f615019) complete. Timing: real 0.303s user 0.039s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":15233,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":5582,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10766,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":676,"thread_start_us":21557,"threads_started":2,"wal-append.queue_time_us":302}
I20240502 08:21:09.683630 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling UndoDeltaBlockGCOp(1f07b2caf1754c469a8e33382f615019): 4838 bytes on disk
I20240502 08:21:09.684654 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: UndoDeltaBlockGCOp(1f07b2caf1754c469a8e33382f615019) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240502 08:21:09.718712 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:09.740973 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.743131 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.790807 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:09.801096 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.805142 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.809418 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:09.824172 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:21:09.878114 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling FlushMRSOp(5c3f0742355442af8a29b971921b7d45): perf score=0.033355
I20240502 08:21:09.945514 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling FlushMRSOp(c6cd36cfde5e480cad86a046d59e8940): perf score=0.033351
W20240502 08:21:09.969033 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:21:09.996295 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: FlushMRSOp(5c3f0742355442af8a29b971921b7d45) complete. Timing: real 0.118s user 0.066s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1192,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":944,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7753,"threads_started":1,"wal-append.queue_time_us":6547}
I20240502 08:21:09.999119 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling UndoDeltaBlockGCOp(5c3f0742355442af8a29b971921b7d45): 4925 bytes on disk
I20240502 08:21:10.004678 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: UndoDeltaBlockGCOp(5c3f0742355442af8a29b971921b7d45) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
W20240502 08:21:10.034966 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 08:21:10.037271 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:21:10.041659 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: FlushMRSOp(c6cd36cfde5e480cad86a046d59e8940) complete. Timing: real 0.096s user 0.047s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":694,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2727,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6409,"threads_started":1,"wal-append.queue_time_us":6307}
I20240502 08:21:10.044255 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling UndoDeltaBlockGCOp(c6cd36cfde5e480cad86a046d59e8940): 4840 bytes on disk
I20240502 08:21:10.044924 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: UndoDeltaBlockGCOp(c6cd36cfde5e480cad86a046d59e8940) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
W20240502 08:21:10.059146 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:21:10.082563 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:10.086768 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 08:21:10.100028 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:21:10.102540 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:10.107997 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:10.127271 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:21:10.138093 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:10.190884 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:10.193571 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:10.201354 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:10.201959 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:10.204500 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:10.220647 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:21:10.233299 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:10.259279 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling FlushMRSOp(1fba813ac35c4382a0917604c2057496): perf score=0.033347
I20240502 08:21:10.282239 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:10.356060 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: FlushMRSOp(1fba813ac35c4382a0917604c2057496) complete. Timing: real 0.097s user 0.026s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":704,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":923,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2961,"threads_started":1,"wal-append.queue_time_us":3052}
I20240502 08:21:10.358373 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling UndoDeltaBlockGCOp(1fba813ac35c4382a0917604c2057496): 4840 bytes on disk
I20240502 08:21:10.359683 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: UndoDeltaBlockGCOp(1fba813ac35c4382a0917604c2057496) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240502 08:21:10.363406 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:10.458612 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:10.561192 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:10.605350 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:10.638232 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:10.664537 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:10.676651 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:10.706045 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:10.716508 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:10.718011 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:10.766039 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:10.795775 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:10.818600 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:10.825721 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:10.888758 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:10.954846 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:10.956454 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:10.960635 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.044581 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.067925 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.069556 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:11.091241 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:11.126637 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:21:11.134743 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:11.139253 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:11.144968 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:11.176331 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:11.176910 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:11.182060 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:21:11.190135 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:11.212651 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:11.220362 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.227550 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.247074 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:11.331650 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.387209 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.403687 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.407259 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.497059 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:11.561069 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.565304 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.613818 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:11.616194 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:11.616858 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:11.622483 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:11.630362 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:11.638794 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:21:11.640847 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:11.646368 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:11.691995 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:21:11.701987 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:11.704461 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:11.750908 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.797310 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.834403 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.863548 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.873615 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.878690 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.990108 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:11.995350 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:12.002884 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:12.009788 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:12.090646 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:12.231622 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:21:12.275883 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:12.284330 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:12.312175 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:12.321702 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:21:12.337877 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:12.354806 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:12.355230 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:12.356634 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:12.369810 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:12.396477 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:12.437134 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:12.461690 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:12.467898 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:21:12.475379 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:12.489405 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:12.525184 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:12.636797 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:12.682081 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:12.683990 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:12.709785 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:12.739449 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:12.767807 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:12.775871 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:21:12.841825 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:12.849496 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:12.849970 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:12.877986 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:12.886204 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:12.889309 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:12.909400 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:12.915596 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:12.950371 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.111636 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.167969 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:21:13.266029 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:13.291359 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:21:13.293000 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:13.327091 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:13.331559 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:13.338429 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:13.351307 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:13.359103 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:13.402514 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.411942 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:13.444013 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.471302 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.479058 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.528246 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.566191 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.581701 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.590108 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.609331 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.662694 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.705191 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.707106 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:13.711392 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.728037 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:21:13.758728 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:13.803920 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:13.818673 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:13.849169 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:13.869180 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:13.922194 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:13.926851 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:13.948801 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:13.961612 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:13.991087 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.021221 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:14.080646 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.093458 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:14.123943 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.125728 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.189399 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.212798 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.213858 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:14.239401 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.282660 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.283703 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:21:14.306568 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:14.333143 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:14.334430 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:14.338856 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:14.356113 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:14.357551 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:21:14.360028 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:14.423820 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:21:14.425033 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:14.466511 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:14.487790 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.556679 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:21:14.557778 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240502 08:21:14.575740 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:21:14.598281 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.710800 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:21:14.766574 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.786512 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.789062 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.792198 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.828580 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:14.836154 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:14.871101 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:14.898917 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:14.939442 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:21:14.996924 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:14.998345 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:15.019786 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:15.049793 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:15.050796 13058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.6:44983, user_credentials={real_user=slave}} blocked reactor thread for 51107.8us
W20240502 08:21:15.058621 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:15.061105 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240502 08:21:15.082087 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:15.089303 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 08:21:15.098448 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:15.109376 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:15.118602 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:21:15.133280 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:15.138583 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:15.153765 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:21:15.165917 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:15.174507 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:15.273061 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:15.286530 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:15.337739 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:15.388438 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:15.461903 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:15.486620 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:21:15.567919 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:15.583762 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:15.619709 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:15.643991 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:15.654688 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:15.656991 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:15.687541 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:15.776119 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:15.792984 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:15.796694 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:15.815474 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:15.918377 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:15.939508 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:15.966265 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:15.970260 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.011866 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:16.014920 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:21:16.016228 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:16.031677 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:16.042749 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:21:16.051437 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:16.051486 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:16.052137 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:21:16.080343 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:16.098697 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:16.137492 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:16.143743 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.185272 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.207883 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:16.243062 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:16.322070 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.442441 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.453130 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.486187 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:21:16.496982 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:16.504801 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:16.517093 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:21:16.517274 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:16.550522 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:16.561770 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:16.563237 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:16.577878 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.596329 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:21:16.603755 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:16.608397 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:16.632669 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.648648 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:21:16.692433 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.725916 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.752859 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.758878 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:16.792671 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.828642 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:16.850397 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:21:16.980206 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:17.004818 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:21:17.010763 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:17.014519 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:17.017532 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:17.026739 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:17.026788 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:17.035416 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:17.035924 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:17.089695 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:17.129726 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:17.146601 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:17.192018 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:17.198446 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:17.330545 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:17.423651 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:17.457633 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:17.462061 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:17.492705 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:17.517880 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:17.583909 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:17.605913 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:17.611900 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:17.653106 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:21:17.661643 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:17.674510 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:17.708354 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:17.708513 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:21:17.740737 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:17.745177 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:17.763127 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:21:17.766767 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:17.798992 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:17.808619 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:17.840519 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:17.874300 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:18.013790 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:18.015240 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:18.123785 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:18.153945 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:18.155453 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:18.184895 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:18.185964 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:18.210073 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:18.222098 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:18.244899 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:18.244988 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:18.263247 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:18.302413 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:18.360466 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:18.374375 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:18.411947 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:18.444396 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:18.466539 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:18.633092 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:18.644822 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:21:18.664494 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:18.667742 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:18.689180 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:18.712065 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:18.726703 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:18.730511 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:18.733819 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:18.760387 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:18.772653 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:18.787447 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:18.797751 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:18.831703 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:18.850782 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:18.996521 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.052002 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.055888 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.075237 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:19.093649 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:19.109782 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:19.144189 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:19.151140 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:19.153237 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:19.177588 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:19.194169 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:19.199522 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.199713 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.222889 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.235152 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:19.247153 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:19.309049 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.310914 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:21:19.329881 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.369737 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:19.384560 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:19.446838 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.467473 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.545807 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.548528 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.576056 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:21:19.582185 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 131 similar messages]
I20240502 08:21:19.629786 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:19.645056 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:19.660553 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:19.671376 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:19.682477 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:19.691736 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:19.715399 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:19.716931 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.731500 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:21:19.755353 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:21:19.765991 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:19.774507 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:19.837011 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:19.838095 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.878790 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:19.881561 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:19.895162 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:20.011329 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:20.013175 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:20.044725 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:20.092617 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 08:21:20.127630 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 08:21:20.135180 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:20.142848 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:20.152086 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:20.165195 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:20.170302 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:20.170655 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:21:20.211688 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:20.287806 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:20.304601 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:21:20.330385 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:20.341643 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:20.355278 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:20.371598 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:20.372601 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:20.397835 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:20.422142 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:20.425899 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:20.436600 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:20.462981 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:20.500319 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:20.501179 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:20.658480 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:21:20.662791 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling FlushMRSOp(6b40f3044cd6485ca72b05873dc93f7e): perf score=0.033336
W20240502 08:21:20.691184 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:20.744002 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:20.748165 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: FlushMRSOp(6b40f3044cd6485ca72b05873dc93f7e) complete. Timing: real 0.085s user 0.064s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":309,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1265,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":719,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6282,"threads_started":2,"wal-append.queue_time_us":6182}
I20240502 08:21:20.750418 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling UndoDeltaBlockGCOp(6b40f3044cd6485ca72b05873dc93f7e): 4840 bytes on disk
I20240502 08:21:20.751082 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: UndoDeltaBlockGCOp(6b40f3044cd6485ca72b05873dc93f7e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240502 08:21:20.899191 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:20.934484 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:20.946218 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:20.977329 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:20.996505 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:21.019975 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:21.024619 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:21.027858 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:21.038846 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:21.051043 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:21.055543 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:21.072479 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:21.076776 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:21.140549 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:21.141806 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:21.142818 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:21.146111 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:21.178202 15687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.4:32979, user_credentials={real_user=slave}} blocked reactor thread for 134197us
W20240502 08:21:21.239439 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:21.277485 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:21.359799 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:21.477622 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:21.480033 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:21.485003 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:21.517100 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:21.538787 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:21.555349 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:21.566861 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:21.573385 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:21.576764 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:21.577570 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:21.579825 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:21.592507 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:21.617938 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:21.618281 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:21.634689 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:21.655334 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:21.731125 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:21.733834 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:21.744666 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:21.971087 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:22.010131 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:22.019532 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:22.039947 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:21:22.056473 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:22.057576 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:22.078286 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:22.080150 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:22.088405 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:22.103101 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:22.127239 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:22.144316 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:22.160051 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:22.160293 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:22.161530 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:22.223702 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:22.285499 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:22.287508 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:22.358635 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:22.365304 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:22.411322 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:22.466326 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:22.505458 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:22.531764 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:22.569038 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:22.582182 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:22.590647 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:22.628244 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:22.630975 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:22.654045 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:22.660697 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:22.670589 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:22.679658 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:21:22.682865 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:22.692268 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:22.711866 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:21:22.722862 13056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.5:44131, user_credentials={real_user=slave}} blocked reactor thread for 48957.2us
W20240502 08:21:22.766022 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:22.774571 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:22.905398 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:22.933143 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:22.974098 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:23.035986 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:23.102130 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:23.151266 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:23.187538 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:23.272357 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:21:23.317831 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:23.339891 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:23.349491 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:23.356698 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:23.370276 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:21:23.375854 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:23.384027 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:23.384917 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:23.404332 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:23.425648 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:23.494433 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:23.533448 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:23.570722 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:23.572269 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:23.622406 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:23.648841 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:23.696606 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:23.714015 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:23.730088 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:23.808602 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling FlushMRSOp(9366b55ca7dc48bfa5d2ed650d7a30e8): perf score=0.033334
I20240502 08:21:23.854660 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:23.877914 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:21:23.878862 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:23.894125 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: FlushMRSOp(9366b55ca7dc48bfa5d2ed650d7a30e8) complete. Timing: real 0.085s user 0.026s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4517,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1463,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":905,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10672,"threads_started":2,"wal-append.queue_time_us":6144}
I20240502 08:21:23.896713 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling UndoDeltaBlockGCOp(9366b55ca7dc48bfa5d2ed650d7a30e8): 4840 bytes on disk
I20240502 08:21:23.899120 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: UndoDeltaBlockGCOp(9366b55ca7dc48bfa5d2ed650d7a30e8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 08:21:23.906440 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:23.911113 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:23.917119 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:23.918315 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:23.919284 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:21:23.961876 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:23.962617 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:23.965848 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:23.995726 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling FlushMRSOp(21f25468f1b541dc9ede27359acc2b9c): perf score=0.033351
W20240502 08:21:24.042248 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:24.054790 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:21:24.061781 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: FlushMRSOp(21f25468f1b541dc9ede27359acc2b9c) complete. Timing: real 0.066s user 0.037s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":17164,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":1581,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":845,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":23051,"threads_started":2,"wal-append.queue_time_us":6060}
I20240502 08:21:24.064272 15934 maintenance_manager.cc:419] P 10f0e36203914b85b77e78609ffced98: Scheduling UndoDeltaBlockGCOp(21f25468f1b541dc9ede27359acc2b9c): 4925 bytes on disk
I20240502 08:21:24.074656 15823 maintenance_manager.cc:643] P 10f0e36203914b85b77e78609ffced98: UndoDeltaBlockGCOp(21f25468f1b541dc9ede27359acc2b9c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240502 08:21:24.077745 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:24.086858 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:24.234371 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:24.276870 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:24.340274 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:21:24.351054 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:24.351744 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:24.361917 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:24.392006 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:24.407176 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling FlushMRSOp(e249007ce4284fd2bc026ab9d6c5d4cb): perf score=0.033403
I20240502 08:21:24.419275 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:24.448221 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:24.467852 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:24.483057 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:24.507486 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:24.515875 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:24.540225 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:24.590552 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240502 08:21:24.591750 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:24.595142 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:24.604934 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:24.605437 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:21:24.613351 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: FlushMRSOp(e249007ce4284fd2bc026ab9d6c5d4cb) complete. Timing: real 0.206s user 0.080s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":46290,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":2364,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6914,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":51868,"threads_started":2,"wal-append.queue_time_us":5578}
I20240502 08:21:24.615761 15534 maintenance_manager.cc:419] P db7893bb29a047b3b82cc60bfd8930a3: Scheduling UndoDeltaBlockGCOp(e249007ce4284fd2bc026ab9d6c5d4cb): 4586 bytes on disk
I20240502 08:21:24.616372 15423 maintenance_manager.cc:643] P db7893bb29a047b3b82cc60bfd8930a3: UndoDeltaBlockGCOp(e249007ce4284fd2bc026ab9d6c5d4cb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240502 08:21:24.643587 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:24.644663 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:24.647212 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:21:24.687600 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:24.768316 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:24.815306 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:21:24.827334 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:24.833900 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:21:24.875330 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:24.877288 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:21:24.905220 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:24.915681 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:24.942160 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:24.958258 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:24.962641 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:24.983975 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:24.987130 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:25.081625 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:25.089257 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:25.092698 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:25.145622 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:25.148684 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:25.175508 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:21:25.176299 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:25.179952 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:25.199316 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:25.211213 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 08:21:25.212404 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:21:25.241029 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:25.281527 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 08:21:25.282423 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:25.297482 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:25.345016 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:25.385289 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:25.406663 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:25.422453 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:25.425025 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:25.444090 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:25.490556 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:25.514958 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:25.528556 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:25.544981 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:25.614331 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:25.647529 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:25.713871 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:25.804471 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:25.851200 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:25.868208 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:25.882042 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:25.997234 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:26.032318 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:26.033157 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:26.037868 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:26.112309 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:26.118382 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:26.118499 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:26.139604 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:26.140419 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:26.144526 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:26.168934 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:26.180073 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:26.187862 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:26.202430 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:26.253604 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:26.258708 12577 maintenance_manager.cc:419] P 17be716fc99c4d588d7726089a2116d2: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033393
I20240502 08:21:26.275589 12576 maintenance_manager.cc:643] P 17be716fc99c4d588d7726089a2116d2: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.004s sys 0.003s Metrics: {"bytes_written":4411,"delete_count":0,"lbm_write_time_us":506,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":9626,"threads_started":1,"update_count":66,"wal-append.queue_time_us":2171}
W20240502 08:21:26.279543 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:26.283201 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:26.302177 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:26.452497 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:26.461341 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:26.521790 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:26.544132 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:26.544471 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:26.580901 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:26.582648 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:26.594785 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:21:26.600337 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:26.605446 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:26.608603 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:26.658836 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:26.717527 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:26.745416 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:26.760855 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:26.783365 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:26.786247 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:26.982921 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:26.992853 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:27.001405 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:27.011054 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:27.015118 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:27.044394 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:27.045428 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:27.086357 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:27.099536 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:27.102063 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:27.119045 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:27.143152 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:27.144459 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:27.238588 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:27.316007 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:27.329133 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:27.335477 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:27.346756 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:27.365700 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:27.373538 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:27.374727 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:27.374820 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:27.381254 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:27.399235 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:27.436658 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:27.441023 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:27.441618 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:27.456002 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:21:27.490907 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:27.510452 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:21:27.525363 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:27.603436 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:27.659693 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:27.710479 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:27.766764 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:27.790891 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:21:27.808288 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:27.823459 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:21:27.872974 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:27.891616 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:27.892385 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:27.900664 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:27.911794 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:27.915927 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:27.940491 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:21:27.951612 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:27.983577 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:21:27.985072 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:28.002880 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:28.016482 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.107033 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.128266 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.240468 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:28.283551 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.347460 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.408730 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:28.468784 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.482646 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.490634 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.495287 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.531364 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:28.536598 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.556912 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.569343 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.628528 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:21:28.659312 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:28.681370 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:28.692694 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:28.694526 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:28.717319 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:21:28.729626 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:28.772380 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:28.798575 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:28.803058 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:28.865895 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:28.879642 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.913838 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:28.981271 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:29.098181 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:29.101338 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:29.103571 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:21:29.155843 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:29.164405 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:29.193392 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:29.225178 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:29.277599 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:29.290620 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:21:29.300733 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:29.310657 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:29.318698 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:29.404937 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:29.418529 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:29.448311 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:29.515076 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:29.610288 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 08:21:29.612145 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:29.645565 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:21:29.671792 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:29.671875 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:21:29.684012 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:29.693450 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:29.736003 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:21:29.740500 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:29.785604 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:29.810546 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:29.824472 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:29.846567 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:29.853932 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:29.864346 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:29.865437 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:21:29.871582 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:29.871943 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:29.898216 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:21:29.910176 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:29.910291 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:29.917861 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:29.946323 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:30.012569 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:30.025074 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:21:30.069790 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:30.180349 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:21:30.192251 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:30.213511 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:30.217947 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:30.223122 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:30.234203 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:30.238121 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:21:30.242868 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:30.257735 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:30.290923 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:30.301119 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 08:21:30.312577 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:30.353300 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:30.390301 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 08:21:30.394766 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:21:30.403086 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:30.405536 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:30.472687 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:30.504243 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:30.563418 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:30.564568 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:30.577474 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:30.625150 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:21:30.649071 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:30.650998 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:30.667687 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:30.668282 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:30.681152 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:30.688558 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:30.699725 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:30.721160 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:21:30.724645 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:30.738960 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:30.767515 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:30.767648 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:30.833007 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:30.843199 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:30.853929 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:30.972100 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:31.032959 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:31.034530 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:31.069053 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:31.205176 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:31.290956 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:31.295500 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:31.301972 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:31.332542 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:31.334895 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:31.355613 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:31.359292 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:31.385211 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:31.410591 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:31.422966 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:21:31.455494 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:31.468663 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:21:31.476472 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:31.485033 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:31.488401 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:31.525485 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:31.684131 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:31.753671 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:31.776829 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:31.794670 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:31.885550 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:31.894496 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:21:31.912247 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:31.932065 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:31.949077 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:31.970697 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:31.975673 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:31.977448 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:31.984120 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:32.023221 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.040493 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.154875 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.165310 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:32.168336 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.190248 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.200237 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:32.306535 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.339679 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.377821 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:32.381489 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:32.392071 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:32.396160 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:32.398749 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:32.414572 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:32.424369 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:32.433147 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:32.439538 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:21:32.480484 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:32.498503 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.499859 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.600790 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:32.683266 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.705299 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.771906 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:32.781905 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.789355 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:32.837661 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:32.897001 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:32.943185 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:32.957197 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:32.960152 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:32.960335 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:32.969880 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:32.970332 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:32.983139 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:32.986162 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:32.993201 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:33.012683 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:33.022572 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:21:33.056033 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:33.123279 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.180648 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:21:33.201869 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:33.346354 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.376284 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.398593 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.402328 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:33.404122 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:33.431017 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:33.441881 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:33.449087 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:33.487725 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.513114 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.554239 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:33.567915 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:33.596688 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:33.606876 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.609992 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.610546 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:33.616312 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:33.666363 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:33.727162 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:33.771104 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.804476 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.808235 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.848811 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:33.851073 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.877733 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:33.947526 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:33.968145 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:34.008764 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:34.124383 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:34.158914 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:34.159519 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:34.175669 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:34.199635 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:34.214289 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:34.220880 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:34.253783 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:34.269701 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:34.283562 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:34.308557 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:34.309566 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:34.331616 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:21:34.383921 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:34.530668 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:34.554553 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:34.604017 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:34.605912 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:34.613875 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:34.631145 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 127 similar messages]
I20240502 08:21:34.640383 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:34.665814 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:34.674072 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:34.682193 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:34.727373 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:34.732323 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:21:34.755964 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:34.762642 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:34.768882 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:34.789047 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:34.816290 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:34.854296 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:34.921262 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:34.963184 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:21:34.965103 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:35.036228 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:35.041415 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:35.096328 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:21:35.120057 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:35.159361 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:35.194536 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:35.195813 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:35.241986 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:35.243093 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:35.257212 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:21:35.263182 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:35.265738 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:35.270671 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:35.271546 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:35.274919 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:35.295241 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:35.316151 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:35.319310 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:35.353374 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:35.424052 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:35.432165 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:35.445384 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 08:21:35.470055 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 08:21:35.470873 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:35.496814 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:35.501775 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:21:35.704246 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:35.728857 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:21:35.741484 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:35.754838 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:35.788687 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:35.795693 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:35.806927 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:35.835826 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:35.848587 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:35.856408 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:35.856794 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:35.870266 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:35.873092 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:35.911002 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:35.914376 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:36.093827 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:36.127091 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:36.136981 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:36.143934 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:36.156121 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:36.164870 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:36.224937 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:36.238726 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:36.245728 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:36.250387 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:36.251046 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:36.263105 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:36.271214 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:36.293663 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:36.295680 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:36.301512 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:36.311936 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:36.335158 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:36.391734 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:36.430521 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:36.555169 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:36.569171 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:36.591497 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:36.597357 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:36.611203 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:36.648959 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:36.839255 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:36.920590 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:36.927198 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:36.940999 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:36.950235 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:36.955979 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:36.968112 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:36.974756 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:37.003155 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:37.016145 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:37.016979 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:37.025743 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:37.045138 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:37.046625 13056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 78289.9us
W20240502 08:21:37.055789 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:37.092284 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:37.098161 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:37.121789 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:37.186959 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:37.233954 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:37.298978 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:37.363626 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:21:37.395267 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:37.414775 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:37.421202 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:37.434612 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:37.441687 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:37.462298 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:37.485296 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:37.537097 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:37.548350 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:21:37.560014 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:37.561061 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:37.660225 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:37.669149 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:37.700399 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:37.717495 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:37.834805 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:37.884132 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:37.916363 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:37.917306 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:37.921672 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:37.960269 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:37.964632 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:37.964650 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:37.964933 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:37.965013 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:21:37.965721 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:38.049990 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:38.054179 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.164884 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.196848 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:38.210525 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.223650 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:21:38.319626 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.335788 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:38.344288 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.349797 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.367564 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.371945 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:21:38.402945 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:38.421726 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:38.429679 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:38.443815 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:38.444223 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:38.498263 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:38.502709 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:38.509649 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.537813 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:38.546912 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:38.557065 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.637460 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:38.642316 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.643132 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:38.693256 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.756078 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.758421 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.817235 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.829980 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:38.840344 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:21:38.948742 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:38.949275 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:38.983882 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:21:38.995034 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:39.015303 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:39.037163 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:39.043256 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:39.046945 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:21:39.068832 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:39.091267 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:39.211318 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:39.255254 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:39.368009 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:39.380915 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:39.394846 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:39.418308 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:39.423584 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:39.430421 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:39.440135 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:39.512668 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:39.644544 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240502 08:21:39.672691 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:21:39.682045 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:39.727514 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:39.738903 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 77 similar messages]
I20240502 08:21:39.789165 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:39.804795 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:39.815644 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:39.817049 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:39.823335 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:39.850673 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:39.853081 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:39.853559 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:39.866722 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:39.882266 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:39.940501 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:39.950510 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:39.993151 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:40.052657 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:21:40.102035 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:21:40.102994 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:40.129819 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:40.146015 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:40.206364 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:21:40.259591 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:40.263708 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:40.285241 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:40.286614 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:40.298156 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:40.298702 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 08:21:40.327462 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:40.347455 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:21:40.353858 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:40.356488 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:40.362599 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:40.387948 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:21:40.411499 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:40.486907 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:40.536748 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 08:21:40.597460 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 08:21:40.598739 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:40.628603 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:40.630736 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:40.648070 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:40.676584 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:40.709412 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:40.720901 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:40.749487 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:21:40.762940 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:40.769523 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:40.780818 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:40.797833 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:40.816104 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:40.818331 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:40.829277 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:40.833854 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:40.880016 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:40.900687 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:21:40.904152 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:40.908416 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:40.922159 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:40.924258 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:40.977749 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:41.013447 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:41.063797 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:41.156313 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:41.163774 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:41.199048 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:41.239598 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:41.252736 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:41.273180 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:41.279991 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:41.280584 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:41.292366 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:41.307034 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:41.317034 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:41.422322 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:41.431759 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:41.433053 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:41.483249 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:41.585281 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:41.623505 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:41.639540 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:41.699702 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:41.709796 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:41.719388 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:41.734002 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:41.740541 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:41.749107 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:41.775224 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:41.775805 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:41.877295 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:41.907907 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:41.910682 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:41.965801 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:41.969588 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:21:41.974092 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:42.010445 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:42.063057 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:42.074160 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:42.112749 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:42.147934 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:42.258477 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:42.326191 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:42.378520 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:42.424350 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:42.452365 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:42.452914 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:42.456507 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:42.483436 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:42.496642 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:42.498764 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:42.503332 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:42.563547 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:42.568857 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:42.591668 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:42.602555 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:42.617184 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:42.634531 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:42.665467 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:42.777140 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:42.826198 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:42.878216 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:42.900421 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:42.951777 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:42.975553 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:42.983647 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:42.984622 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:43.021353 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.041770 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:43.047183 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:43.068838 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:43.085129 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:43.115550 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:43.118219 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:43.123169 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:21:43.143481 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:43.183030 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.219944 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.222545 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.269166 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:43.290243 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.307168 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:43.309942 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.371301 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.445055 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:43.474340 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:43.482522 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:43.507508 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.518764 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:21:43.526219 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:43.542551 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:43.558046 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:43.567940 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:43.576352 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:43.593184 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:43.608289 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:43.674643 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.675314 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.722311 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:43.777004 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:43.806418 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.857661 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.858392 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:43.903703 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:43.958878 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:43.984517 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:43.989111 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:44.001080 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:44.003357 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:44.012665 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:44.019071 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:44.047040 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:44.054831 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:44.183969 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:44.236605 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:44.290241 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:44.338162 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:44.356653 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:44.410219 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:44.436688 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:44.449903 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:44.450397 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:44.450944 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:44.480295 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:44.495002 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:44.519593 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:44.530625 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:44.556012 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:44.564299 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:44.580194 13057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.8:46303, user_credentials={real_user=slave}} blocked reactor thread for 110155us
W20240502 08:21:44.609658 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:44.667557 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:44.685058 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 08:21:44.728909 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:44.761282 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:44.762909 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:44.770876 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:21:44.771589 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:44.796077 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:44.923550 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:44.944294 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:44.965175 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:45.009472 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:45.068497 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:45.083009 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:45.096616 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:21:45.115830 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:45.151757 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:45.183822 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:45.197407 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:21:45.201711 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:45.203313 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:45.228400 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:45.257679 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:45.259002 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:45.259554 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:45.287637 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:45.313661 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:45.326035 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 08:21:45.388346 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:21:45.400553 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:45.408668 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:45.434438 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:45.466441 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:45.484596 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:45.489315 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:45.507939 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:45.604975 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 08:21:45.606189 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:45.661789 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:45.665834 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:45.676234 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:45.700851 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:45.714516 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:45.741730 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 08:21:45.748437 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:45.764915 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:45.766436 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:45.766849 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:45.800904 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:45.823032 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:45.829764 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:45.830986 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:45.855991 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:45.868249 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:45.910372 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:45.965713 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:46.075610 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:46.132794 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:46.180176 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:46.203979 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:46.218278 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:46.232203 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:46.234246 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:46.244649 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:46.257407 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:46.261186 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:46.264207 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:46.276751 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:46.299988 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:46.300673 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:46.335451 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:46.348626 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:46.402904 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:21:46.467198 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:46.567448 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:46.655105 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:46.666869 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:46.677321 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:46.678572 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:46.702600 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:46.740746 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:46.759876 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:46.769838 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:46.799351 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:46.831588 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:46.894699 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:46.918300 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:46.922040 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:46.927673 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:46.962520 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:47.007678 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:47.119678 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:47.186517 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:47.187060 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:47.188107 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:47.203172 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:47.203442 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:47.211462 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:47.221515 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:47.224969 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:47.230299 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:47.238550 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:47.242691 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:47.245321 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:47.271374 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:47.345592 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:47.383282 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:47.412405 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:47.544275 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:47.628981 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:47.672211 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:47.691846 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:47.712265 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:47.740422 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:47.775607 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:47.776082 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:47.782332 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:47.874931 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:47.880852 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:47.882750 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:47.889710 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:47.895231 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:47.897514 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:47.907208 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:21:47.912777 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:47.920987 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:47.933562 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:47.938793 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:47.987386 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:47.997783 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:48.020193 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:48.045899 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:48.161746 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:48.172302 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:48.176023 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:48.281126 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:48.282766 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:48.355758 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:48.356331 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:48.359725 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:48.368089 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:48.382535 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:48.416996 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:48.417755 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:48.451815 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:48.480259 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:48.528033 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:48.639371 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:48.642176 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:48.671195 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:48.752048 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:48.756356 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:48.796797 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:48.802008 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:48.813851 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:48.892750 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:48.901631 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:48.928258 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:48.944305 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:48.945261 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:48.953912 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:48.962123 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:48.979061 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:48.995529 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:49.027988 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:49.042346 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:21:49.048818 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:49.128674 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:49.131956 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:49.200842 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:49.209098 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:49.285699 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:49.332470 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:49.397934 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:49.400933 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:49.403164 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:49.404549 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:49.436996 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:49.461880 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:49.465961 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:49.523866 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:49.531903 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:49.544245 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:49.557512 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:49.654839 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:49.674710 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:49.692972 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240502 08:21:49.728951 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:49.825229 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240502 08:21:49.827173 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:49.847188 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:49.899473 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:49.901643 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:49.927548 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:49.931737 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:49.946075 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:49.958362 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:49.959674 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:49.966434 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:49.973080 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:49.985563 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:50.051649 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:50.068029 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:50.113148 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:50.140316 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 08:21:50.141216 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:50.146852 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:50.234375 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:50.262512 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:21:50.282964 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:50.286983 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:50.308979 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:50.324849 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:21:50.352367 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:50.355720 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:50.373123 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:50.379379 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:50.395642 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:21:50.401687 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:50.416793 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:50.437188 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:50.487228 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:50.514050 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:50.580744 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:50.612275 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 08:21:50.652793 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:50.675263 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:50.682677 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:50.695360 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:50.701356 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:50.727703 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:50.741698 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:50.763361 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:50.774845 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:50.807228 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:50.819459 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:50.914160 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 08:21:50.985180 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:50.989195 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:51.025082 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:51.056327 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:51.151527 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:51.154119 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:51.155743 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:51.159278 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:51.174726 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:51.182664 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:51.184790 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:51.192184 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:51.209854 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:51.234603 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:51.236143 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:51.254921 13057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.8:46303, user_credentials={real_user=slave}} blocked reactor thread for 59790.4us
W20240502 08:21:51.278904 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:51.280009 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:51.291234 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:51.299301 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:51.386242 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:51.476629 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:51.537055 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:21:51.596503 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:51.620481 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:51.625382 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:51.671231 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:51.671360 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:51.704595 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:51.735592 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:51.735616 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:51.744711 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:51.757673 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:51.788758 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:51.829584 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:51.836241 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:51.836536 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:51.868561 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:51.904692 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:52.087182 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:52.098038 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:52.106334 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:52.146046 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:52.158010 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:52.182899 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:52.187410 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:52.193940 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:52.199899 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:52.213762 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:52.220543 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:52.234234 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:52.263866 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:52.307439 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:52.361265 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:52.471203 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:52.580962 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:52.585924 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:52.589394 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:52.615367 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:52.627004 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:52.630678 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:52.661808 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:52.668591 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:52.671674 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:52.700148 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:52.701355 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:52.701601 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:52.701719 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:52.714576 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:52.735124 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:52.735133 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:52.736192 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:52.784214 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:52.784999 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:52.807356 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:52.807965 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:52.841115 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:52.883517 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:52.900219 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:21:52.924567 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:52.971527 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:52.994414 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:53.022310 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:53.061998 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:53.211572 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:53.254398 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:53.289415 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:53.319453 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:53.356375 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:53.369494 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:53.369678 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:53.457582 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:53.485759 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:53.496245 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:53.571545 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:53.585245 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:53.649008 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:53.660141 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:53.676792 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:53.678577 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:53.690392 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:53.837810 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:53.858013 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:53.875962 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:53.938143 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:53.943790 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:53.956137 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:53.964083 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:53.970726 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:53.976815 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:53.998533 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:54.015084 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:54.065887 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:54.105257 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:54.183223 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:54.198114 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:54.222262 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:54.236742 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:54.286152 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:54.291313 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:54.412690 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:54.420782 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:54.423347 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:54.426036 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:54.444633 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:54.449893 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:54.453900 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:54.490818 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:54.536151 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:54.546165 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:21:54.558176 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:54.636503 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:54.657421 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:54.661366 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:54.673658 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:54.800843 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240502 08:21:54.806202 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:54.831916 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:54.873620 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:54.891667 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:54.926723 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:54.929783 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:54.965318 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:54.974579 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:54.976359 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:21:54.986709 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:54.990674 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:54.997821 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:55.014535 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:55.030853 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:55.071156 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:55.094072 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:55.118117 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:55.141054 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:55.182474 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:55.190315 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:55.192564 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:21:55.193452 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:55.218832 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:55.274405 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:55.300562 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:55.301759 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:21:55.314556 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:55.353585 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:55.391045 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:55.400030 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:55.402066 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:55.454345 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:55.469179 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:21:55.498937 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:21:55.502583 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:55.522269 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:55.528892 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:55.543874 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:55.547250 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:55.562705 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:55.574985 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:55.592532 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:55.601467 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:55.603797 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:55.624128 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:55.762168 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:55.780833 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 08:21:55.781672 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:55.816135 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:55.892922 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:55.959256 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 08:21:56.051441 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:56.052134 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:56.177265 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:56.179842 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:56.197173 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:56.208726 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:56.211030 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:56.222748 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:56.236909 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:56.255615 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:56.277487 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:56.296458 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:56.297035 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:56.314229 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:21:56.314420 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:56.416149 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:56.485823 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:56.566663 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:56.625404 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:56.684731 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:56.688043 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:56.715440 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:56.723089 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:56.723989 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:56.731259 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:56.773746 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:56.795238 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:56.803676 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:56.835925 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:56.850721 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:56.850852 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:56.867569 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:56.870329 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:56.906543 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:56.970448 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:56.979128 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:57.070292 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:57.100109 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:57.178751 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:57.183338 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:57.201519 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:57.202759 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:57.213598 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:21:57.250950 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:57.257105 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:57.288264 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:57.306192 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:57.314193 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:57.386559 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:57.398662 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:57.413255 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:57.481592 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:57.510020 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:57.513621 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:57.532158 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:57.540123 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:57.545920 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:57.551075 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:57.566035 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:57.628875 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:57.670011 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:57.678627 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:21:57.682972 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:57.717813 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:57.718055 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:57.747400 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:57.747802 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:57.755519 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:57.762599 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:21:57.800060 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:57.806604 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:21:57.810582 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:57.985954 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:57.988526 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:58.023806 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:58.052490 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:58.101801 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:58.106629 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:58.109722 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:58.119987 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:21:58.143070 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:58.151674 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:58.162750 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:58.200156 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:58.200202 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:21:58.220255 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:58.233947 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:58.241525 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:58.262223 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:21:58.313277 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:58.318956 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:58.349956 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:58.375649 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:21:58.446605 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:58.457625 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:58.496748 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:58.571002 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:58.605370 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:58.817111 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:21:58.899215 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:58.923519 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:21:58.933254 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:58.948827 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:58.961161 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:58.988279 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:21:59.009522 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:59.020711 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:59.020936 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:59.063270 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:59.066857 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:59.158852 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:59.255445 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:59.262019 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:59.287966 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:59.296092 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:59.308360 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:59.343834 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:59.347270 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:21:59.393885 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:59.398046 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:21:59.400255 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:59.415994 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:59.416070 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:21:59.439579 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:59.479913 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:59.485487 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:21:59.500906 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:59.509897 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:59.547122 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:59.550347 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:21:59.572499 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:59.653136 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:21:59.778964 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:21:59.862711 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:59.870452 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:21:59.872066 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:21:59.873318 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:21:59.878535 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 132 similar messages]
I20240502 08:21:59.894142 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:21:59.898638 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:21:59.909549 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:21:59.927719 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:21:59.945956 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:21:59.946365 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:21:59.952414 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:21:59.956833 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:21:59.967967 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:21:59.992269 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:22:00.022881 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:22:00.119793 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:00.123858 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 08:22:00.126150 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:00.152324 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:00.231410 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:00.257810 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:00.272967 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:22:00.273986 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:22:00.274914 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:22:00.344592 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:00.357586 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:22:00.359599 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:00.363517 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:00.424393 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:00.445710 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:00.474891 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:00.474956 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:00.486584 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:22:00.502735 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:00.542119 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:00.552450 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:00.563125 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:00.580166 12797 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.8:46303, user_credentials={real_user=slave}} blocked reactor thread for 72786.8us
I20240502 08:22:00.581295 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:00.594220 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:00.602970 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:00.628461 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:22:00.629447 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:00.643051 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:00.679884 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:22:00.746230 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:00.766242 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:22:00.817195 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:22:00.822595 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:22:00.848928 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 08:22:00.851310 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:00.870502 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:00.963871 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240502 08:22:00.998023 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:01.005031 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:01.023480 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:01.055212 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:01.059247 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:01.092057 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:01.095036 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:01.122088 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:01.130883 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:01.147943 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:01.205726 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:01.384512 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:22:01.440549 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:01.493669 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:01.621973 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:01.660138 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:01.662197 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:01.717808 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:01.724681 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:01.731109 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:01.745476 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:01.752542 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:01.756816 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:01.774920 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:01.794420 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:01.859633 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:01.871760 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:01.874831 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:01.877547 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:01.897202 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:01.903378 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:02.001601 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:02.113682 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:02.130431 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:02.147830 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:02.168782 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:22:02.205782 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:22:02.253783 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:02.255955 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:02.260581 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:02.279903 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:02.303102 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:02.307183 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:02.313133 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:02.330250 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:02.338662 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:02.345237 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:02.346076 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:02.378703 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:02.442881 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:02.494737 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:02.527590 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:02.538225 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:02.568490 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:22:02.576962 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:02.627120 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:02.702522 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:02.704077 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:02.715714 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:02.721446 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:02.738993 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:02.744810 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:02.769220 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:02.780431 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:02.795198 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:22:02.798418 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:02.807523 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:02.825170 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:22:02.828580 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:02.846580 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:02.904232 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:02.935421 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:02.951301 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:02.978745 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:03.055405 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:03.103417 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:03.118327 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:03.161696 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:03.163969 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:03.174954 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:03.184028 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:03.195438 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:03.252326 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:03.259493 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:03.296785 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:03.297704 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:03.303716 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:03.325491 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:03.338502 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:03.350602 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:03.394588 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:03.407622 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:22:03.567462 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:03.601243 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:22:03.668664 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:03.704874 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:03.714996 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:03.729918 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:03.746604 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:03.766008 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:03.808517 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:03.813146 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:03.857550 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:03.886077 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:03.900732 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:03.994272 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.074149 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.118422 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.142127 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.267503 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.280468 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.280581 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:04.318522 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.339818 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.377552 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.377694 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.383975 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:22:04.439952 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:04.451375 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:04.451946 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:22:04.488308 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:04.513087 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:22:04.544838 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:04.546525 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:22:04.554694 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:04.555235 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:04.603037 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:04.629696 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:04.672711 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:04.673153 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:04.793466 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.803073 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:04.806942 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.834141 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:04.953897 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:22:04.970521 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:04.973511 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:04.997156 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 08:22:04.998514 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:05.006942 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:22:05.010275 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:05.012838 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:05.027007 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:22:05.061089 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:05.061524 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:05.103389 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:05.127987 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:05.140149 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:22:05.140959 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:05.149497 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:05.192855 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:05.231204 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:05.254376 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:05.276239 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:05.276441 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:22:05.278081 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:05.291136 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:05.298688 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:05.312031 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:05.330265 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:05.450690 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:22:05.462565 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:05.472043 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:05.472497 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:05.474550 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:05.498044 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:05.502776 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:22:05.517642 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:22:05.530121 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:05.542436 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:05.580096 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:05.590158 12665 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.8:46303, user_credentials={real_user=slave}} blocked reactor thread for 51026.9us
W20240502 08:22:05.601042 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 08:22:05.608698 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:05.671430 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:05.748478 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:05.803119 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:05.830847 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:05.849470 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 08:22:05.852187 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:05.852231 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:22:05.972278 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:05.984592 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:05.998402 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 08:22:06.003587 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:22:06.007541 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:06.011255 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:06.036890 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:06.066529 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:22:06.067447 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:06.137642 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:06.156153 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:22:06.185258 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:06.203377 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:06.217370 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:06.276764 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:06.293308 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:06.467839 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:06.475050 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:22:06.494993 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:06.517407 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:06.535462 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:06.540971 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:22:06.554086 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:06.558701 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:06.597735 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:22:06.609481 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:06.611217 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:06.634984 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:22:06.638419 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:06.675552 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:06.737465 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:06.791674 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:06.850522 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:06.869467 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:06.894306 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:06.906423 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:06.941504 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:06.973356 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:07.004326 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:07.066588 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:07.166857 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:07.225821 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:07.239517 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:07.264989 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:22:07.271137 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:07.276321 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:07.283046 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:22:07.307528 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:07.316514 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:07.318346 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:07.348980 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:07.362114 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:07.367158 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:07.412449 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:07.446874 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:07.534044 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:07.584277 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:07.593248 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:07.678414 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:07.730598 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:07.734870 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:07.767565 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:07.785977 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:07.786549 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:22:07.812286 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:07.818964 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:07.860081 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:07.864187 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:07.866358 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:07.866559 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:07.881263 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:07.897699 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:07.906953 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:07.916049 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:07.941918 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:07.955410 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:07.972352 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:07.994383 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:08.014760 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:08.171085 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:08.180286 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:08.183084 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:08.291085 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:08.309101 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:08.309290 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:08.314682 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:22:08.363623 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:08.370740 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:08.382342 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:08.385355 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:08.403656 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:08.407608 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:08.410789 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:08.411592 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:08.417963 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:08.472330 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:08.528164 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:08.530493 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:08.552695 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:08.811558 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:08.861245 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:08.875365 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:08.913823 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:08.922659 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:08.927138 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:08.953020 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:08.957808 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:08.958854 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:08.998239 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:09.028766 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:09.049568 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:09.055909 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:09.077633 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:09.107200 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:09.122362 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:09.163009 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:09.173379 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:09.189164 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:09.225930 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:09.330777 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:09.410498 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:09.435907 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:09.466168 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:09.475528 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:09.489826 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:09.492489 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:09.499420 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:09.532532 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:09.556869 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:09.559033 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:09.565722 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:22:09.588871 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:09.608091 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:09.646447 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:09.682161 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:09.718961 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:09.847831 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:09.847935 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:09.895833 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:09.952775 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:09.984342 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:10.010535 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 08:22:10.097074 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:10.097878 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:10.102305 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:10.125591 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:22:10.150391 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:10.158004 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:22:10.160979 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:10.175196 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 08:22:10.178267 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:22:10.195892 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:10.201124 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:10.259497 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:10.278246 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:10.300043 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:10.319295 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:10.323450 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:10.346915 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:10.392011 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:10.400099 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:10.400569 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:10.413100 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:10.422124 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:22:10.423115 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:10.460194 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:10.516985 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:22:10.533252 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:10.619732 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:10.625434 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:10.641789 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:10.720805 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:10.754768 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 08:22:10.758713 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:10.819437 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:10.830399 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:22:10.834656 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:10.851013 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:10.857646 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:10.867053 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 08:22:10.870529 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:10.882321 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:10.898808 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:10.909556 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:10.953357 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:10.971040 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:11.019899 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:11.032732 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:11.059448 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 08:22:11.060379 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:22:11.185333 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:22:11.250725 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:11.276939 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:11.278710 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:11.298559 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:11.340405 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:11.355367 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:22:11.364792 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:11.399331 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:11.411953 15688 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.5:44131, user_credentials={real_user=slave}} blocked reactor thread for 48234.5us
W20240502 08:22:11.415642 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:11.415969 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:22:11.457307 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:11.477416 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:11.487934 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:11.564656 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:11.581336 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:11.649461 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:11.688114 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:11.695294 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:11.702167 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:11.833317 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:11.848863 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:11.851472 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:11.877395 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:11.898726 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:11.903246 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:11.909854 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:11.937888 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:11.944770 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:22:11.972645 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:12.040534 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:12.067829 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:12.160647 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:12.171491 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:12.214879 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:12.245872 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:12.279738 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:12.291210 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:12.294032 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:12.312193 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:22:12.348860 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:12.377351 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:12.400065 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:22:12.423234 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:12.431620 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:12.446513 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:22:12.446843 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:12.469558 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:12.503960 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:12.541886 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:12.569236 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:12.576371 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:12.627416 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:12.636940 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:12.705667 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:12.739699 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:12.762076 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:12.784759 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:12.799345 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:12.812772 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:12.883461 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:12.887472 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:12.919287 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:12.929009 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:12.956682 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:12.965196 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:12.978720 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:22:13.054956 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:13.055512 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:13.103032 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:13.115931 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:13.123263 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:13.138520 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:13.143077 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:13.150779 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:13.189891 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:13.230996 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:13.242276 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:13.296586 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:22:13.304723 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:13.454874 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:13.466590 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:13.466900 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:13.509785 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:13.555049 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:13.586663 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:22:13.589804 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:13.619844 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:13.650928 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:13.651923 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:13.686049 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:13.711529 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:13.741931 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:22:13.770512 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:13.798700 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:14.004170 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:22:14.048028 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:14.064188 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:14.086169 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.086571 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.091063 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:22:14.103596 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:14.110419 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:22:14.123977 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:14.131815 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:14.131847 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:14.146706 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:14.225157 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:14.232188 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:14.243427 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.311952 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.344727 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.404254 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.505192 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:22:14.552145 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:14.558689 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:14.590135 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:14.592934 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:14.606765 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:22:14.607129 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.630079 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:22:14.649441 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:14.710815 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:22:14.715744 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:14.745554 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:14.763353 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:14.767343 12666 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.6:44983, user_credentials={real_user=slave}} blocked reactor thread for 137272us
W20240502 08:22:14.798207 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.806264 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.807049 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.830981 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:14.851848 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.859645 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.938944 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:14.945461 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:14.980159 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:15.015873 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240502 08:22:15.016180 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:15.085872 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:15.099483 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:22:15.119148 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:15.120712 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:15.125927 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:15.161062 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:22:15.162235 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:15.164295 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:15.164474 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:15.171330 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:15.177968 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:22:15.179433 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:22:15.193039 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:15.225289 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:15.265455 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:15.343405 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:15.344426 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:15.344851 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:15.409833 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:15.460085 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:15.466899 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:22:15.467751 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:15.467926 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:15.478332 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:15.481338 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:15.553112 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:15.592962 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:15.599519 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:22:15.781219 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:15.783196 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:15.784965 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:15.794770 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:22:15.850067 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:22:15.864531 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:15.869155 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:15.885258 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 08:22:15.898634 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:15.909960 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:15.932601 13056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 48130.6us
I20240502 08:22:15.941955 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:15.944151 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:22:15.955062 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:15.973250 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:15.999675 12664 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 58185.7us
W20240502 08:22:16.022670 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:16.022902 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:16.024894 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:16.049755 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:16.091481 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:16.307111 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 08:22:16.337515 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:16.357138 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:22:16.378091 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:16.383533 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:16.392495 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:16.419734 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:16.429023 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:16.448796 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:16.462906 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:16.463732 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:16.502929 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:16.592079 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:16.636634 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:16.652582 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:16.724623 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:16.729934 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:16.791177 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:16.891798 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:16.904021 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:22:16.912000 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:16.914094 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:16.916139 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:16.920141 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:16.931550 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:16.931793 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:16.943928 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:16.951644 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:17.014151 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.101539 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.135927 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:17.135955 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:17.140902 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:17.192801 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.200598 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.223363 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.275516 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:17.295653 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:22:17.310528 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:17.321333 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:22:17.327561 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:17.344434 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:17.357544 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:17.365936 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:17.368973 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.383340 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.398386 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:17.399575 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:17.407156 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:17.459370 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:17.484025 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:17.500835 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.575558 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:17.626070 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:17.637696 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.645004 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.692343 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:17.711009 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.732262 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.740780 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.844779 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:17.855515 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:22:17.888509 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:17.921689 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:17.943235 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:17.972818 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:17.977872 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:17.993431 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:18.002007 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:18.007691 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:18.032794 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:18.045178 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:18.125380 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:18.130662 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:18.205160 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:18.234810 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:18.235529 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:18.279461 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:18.358827 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:18.424784 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:18.426362 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:18.459986 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:18.543915 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:18.553233 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:22:18.569315 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:18.619024 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:18.659548 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:18.702628 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:18.717504 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:18.731637 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:18.741528 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:18.760788 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:18.792565 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:18.794446 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:18.885156 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.043264 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.060058 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.062763 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:19.121548 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:19.132654 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:19.174773 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:19.208438 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:22:19.209720 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:19.209784 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:19.210216 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:19.249527 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:19.266233 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:22:19.305239 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:19.383029 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.394503 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.455251 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:19.503815 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.535317 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.554953 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.596168 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:19.613488 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.634297 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:19.689983 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:19.717464 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:22:19.717944 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:19.721704 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.727411 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.735725 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.735944 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:19.765136 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:19.776736 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:19.796648 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:19.808408 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:19.815912 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:19.820361 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:19.841599 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:19.851536 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:19.884261 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:19.902746 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:19.915359 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:19.921367 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:19.937633 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:19.943181 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:19.974505 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:20.017625 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240502 08:22:20.087946 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:20.107077 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:20.221484 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:20.224615 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:20.231094 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:20.235474 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:20.239542 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:20.240115 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:20.240361 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:22:20.252585 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:20.263832 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:20.289525 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:20.304981 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:20.328807 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:20.389338 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:20.403383 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:20.439677 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:20.442093 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:20.454993 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:20.472178 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:20.488914 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:22:20.586513 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:20.617769 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:22:20.638398 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:20.669507 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:20.675356 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:20.697387 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:22:20.722043 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:20.729087 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:20.765018 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:20.786679 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:20.817971 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:22:20.866986 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:20.888227 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:20.888603 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:22:20.892061 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:20.921097 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 08:22:20.922293 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:20.994925 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:21.009214 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:22:21.020068 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:21.026691 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:21.121520 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:21.179008 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:21.249707 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:21.263216 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:21.289764 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:21.373036 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 08:22:21.467931 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:21.470150 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:21.505235 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:21.525758 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:21.526234 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:21.530440 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:21.544247 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:22:21.567147 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:21.621879 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:21.769333 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:21.806159 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:21.806380 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:21.828907 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:21.864316 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:21.871048 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:21.872716 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.003607 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:22.011042 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:22.011454 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:22.013612 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:22.040656 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:22.047770 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:22.047909 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:22.048715 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:22.054023 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:22.064855 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:22:22.096796 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:22.131175 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.132897 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.135349 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:22.228559 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:22:22.231211 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.271481 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.292315 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.305470 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.328894 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:22.410854 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.413848 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.429734 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:22.461949 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:22.467494 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:22:22.476301 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:22.484246 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:22:22.504154 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:22.517701 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:22.525044 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:22.536401 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:22.539691 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:22.548779 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:22.548784 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:22.548817 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.591660 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:22.597582 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:22.634795 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.640990 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.717569 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.720991 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:22.758154 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:22.910804 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:22.973785 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:22.998625 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:22.999591 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:23.008486 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:23.015092 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:23.035601 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:23.061796 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:23.089387 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:23.091609 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:23.102560 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:23.119156 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:23.158195 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:23.200137 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:23.260543 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:23.348886 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:23.358198 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:23.444963 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:23.486541 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:23.509483 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:23.510454 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:23.517658 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:23.521252 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:23.522104 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:23.551873 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:22:23.556290 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:23.577927 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:23.585816 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:23.627130 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:23.636418 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:23.784849 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:23.820226 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:23.825649 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:23.898087 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:23.932150 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:24.121167 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:24.160426 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:24.175698 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:24.195147 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:24.205518 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:24.208823 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:24.209053 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:22:24.223582 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:24.227686 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:24.244880 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:24.250842 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:24.290741 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:24.290978 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:24.326695 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:24.398200 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:24.407691 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:24.468745 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:24.617640 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:22:24.628808 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:24.635583 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:24.639359 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:24.652920 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:24.671506 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:22:24.675985 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:24.703096 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:24.703330 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:24.710114 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:24.726225 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:24.728191 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:24.750450 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:24.768994 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:24.775416 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:24.794710 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:24.809470 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:24.825111 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:24.836097 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:24.852749 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:24.860214 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:24.872556 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:24.887706 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:24.892091 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:24.931188 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:25.029243 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:25.049932 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240502 08:22:25.051206 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:25.078097 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:25.082017 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:25.084883 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:25.126674 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:25.131511 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:22:25.152355 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:25.182880 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:25.187443 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:25.199971 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:25.205814 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:25.226552 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:25.246866 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:25.259125 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:22:25.278122 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:25.351946 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:25.453555 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:25.468215 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:25.508253 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:22:25.509078 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:25.551620 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:25.615410 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:25.653142 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:22:25.660959 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:25.671433 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:25.672791 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:25.675314 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:25.706859 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:22:25.715960 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:25.720669 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:25.728258 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:25.741303 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:25.787307 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:25.856235 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:25.874958 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:25.895645 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:25.981140 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:22:26.007624 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:26.023998 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 08:22:26.073958 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:26.101121 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:26.137578 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:22:26.200814 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:26.212842 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:22:26.213699 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:26.230691 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:26.247637 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:22:26.251734 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:26.255342 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:26.262355 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:26.286484 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:26.314141 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:26.320487 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:26.337397 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:26.629994 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:26.674132 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:26.704849 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:26.748471 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 08:22:26.753600 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:26.765492 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:26.785260 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:26.798110 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:26.842833 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:26.845547 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:22:26.947592 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:26.969312 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:22:26.978996 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:26.979553 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:26.986402 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:27.003365 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:27.005487 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:27.031685 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:27.054591 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:22:27.056180 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:27.136814 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:27.191411 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:27.211076 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.222213 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.226819 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:27.297811 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:27.298331 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.309157 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.320122 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:27.341876 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.371709 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.375082 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.384744 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:22:27.412148 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:27.430292 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:27.434763 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:27.437076 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:27.458225 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.491556 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:22:27.500586 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:27.523042 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:27.525199 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.536160 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:27.573354 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:27.595144 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:27.596282 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.649955 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:27.660319 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.703750 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.710989 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:27.722164 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:27.802927 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:27.885574 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:27.930013 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:27.979693 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:27.981261 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:27.982317 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:27.990262 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:28.011703 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:28.052124 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:28.084597 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:28.085019 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:28.127959 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:28.148272 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:28.151357 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:28.216591 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:28.217756 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:28.284057 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:28.332320 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:28.374537 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:22:28.433413 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:28.445557 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:28.447253 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:28.476600 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:28.487025 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:28.488058 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:28.507517 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:28.507715 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:28.535188 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:28.567656 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:28.615756 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:28.700330 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:28.749907 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:28.797689 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:28.854879 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:22:28.920758 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:22:28.979602 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:29.008469 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:29.038177 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:29.060312 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:29.093165 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:29.094780 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:29.096292 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:29.136394 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:29.143747 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:29.168246 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:29.267159 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:29.275833 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:29.336511 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:29.385107 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:29.402686 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:29.462041 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:29.556414 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:29.613523 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:29.618191 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:29.626672 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:29.627537 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:29.680505 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:29.697250 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:29.718058 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:29.723513 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:29.730944 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:29.734151 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:29.743590 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:29.748476 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:29.775709 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:29.793946 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:29.846766 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:29.853866 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:29.861579 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:29.875146 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:29.876919 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:29.877846 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:29.884159 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:29.912801 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:30.001386 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:30.029893 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:30.089255 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:30.094961 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:30.102146 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:30.110014 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240502 08:22:30.113811 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:30.158252 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:30.174297 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:30.178329 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:30.192302 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:30.235585 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:22:30.255852 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:30.273854 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:30.281281 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:22:30.285578 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:30.287003 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:30.288479 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:30.325191 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:30.344676 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:30.396039 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:30.400660 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:30.456442 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:30.468612 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:30.539456 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:22:30.540422 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:30.565259 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:30.619933 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:30.667474 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:30.670202 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:30.723266 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:22:30.724722 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:30.760922 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:30.772429 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:30.838626 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:22:30.843958 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:30.851220 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:22:30.875763 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:30.898674 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:30.903888 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:30.923238 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:30.934957 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:22:30.952112 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:30.979974 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:22:31.003806 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:31.071580 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 08:22:31.197738 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 08:22:31.199721 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:31.212353 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:31.278846 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:22:31.361773 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:31.381819 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:31.401542 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:31.416669 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:31.460902 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:31.473008 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:31.474714 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:31.474826 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:31.527900 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:31.538769 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:22:31.594839 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:31.680621 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:31.704340 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:31.770516 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:31.786396 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 08:22:31.835207 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:31.849223 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:31.901690 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:22:31.903563 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:31.905613 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:31.906319 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:31.919759 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:31.958521 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:31.974231 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:31.989001 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:32.008646 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:32.030407 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:32.057067 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.057090 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:32.073886 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.106354 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.137275 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:32.162781 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:32.215510 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:32.224143 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.252111 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.260763 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.272044 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:32.272320 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.302054 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.303139 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:32.353750 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.462743 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:32.473413 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.602613 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.638747 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.672892 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.689711 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:32.698335 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:32.730778 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:22:32.732215 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:32.763039 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:32.764564 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:32.794946 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:32.799957 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:32.817252 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:32.823894 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:32.866408 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:32.879887 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:32.910261 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:32.913362 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:32.920979 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:32.937323 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:32.943365 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.944979 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:32.945088 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:32.974675 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:32.992512 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:33.023738 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:33.119119 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:33.127465 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:33.270260 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:33.293908 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:33.308125 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:33.325239 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:33.360585 12926 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.5:44131, user_credentials={real_user=slave}} blocked reactor thread for 57808.3us
I20240502 08:22:33.391829 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:33.423754 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:33.455463 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:33.459182 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:33.459637 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:33.523594 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:33.542028 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:33.574111 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:33.751242 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:22:33.761067 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:33.801852 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:33.806972 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:33.914158 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:33.977820 15419 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.1:33183, user_credentials={real_user=slave}} blocked reactor thread for 74619.2us
I20240502 08:22:34.020218 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:34.036948 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:34.059494 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:34.061245 12664 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.2:36933, user_credentials={real_user=slave}} blocked reactor thread for 52914.5us
I20240502 08:22:34.066558 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:34.099186 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:34.123154 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:34.124212 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:34.148303 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:34.178108 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:34.194288 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:34.242911 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:34.248340 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:34.314713 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:34.331522 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:34.358716 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:34.484439 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:22:34.491891 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:34.503325 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:34.507210 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:34.528558 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:22:34.555016 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:34.560320 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:34.561421 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:34.579087 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:34.585162 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:34.636864 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:34.648612 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:34.707722 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:34.730157 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:34.733601 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:34.734844 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:34.746311 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:34.807448 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:34.811910 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:34.868654 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:34.885859 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:34.897732 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:34.908605 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:34.936914 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:34.994397 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:35.063153 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:35.080920 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:35.086565 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:35.096788 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:35.117580 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 131 similar messages]
I20240502 08:22:35.118419 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:35.132205 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:35.153591 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:35.159837 15418 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.7:38851, user_credentials={real_user=slave}} blocked reactor thread for 51529.5us
W20240502 08:22:35.168339 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:22:35.175016 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:35.188186 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:35.202615 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:35.217202 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:35.240435 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:35.241896 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:35.250360 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:35.291473 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:35.306303 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:35.310714 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:35.353957 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:22:35.391526 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:35.395156 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:35.487958 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:35.504139 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:35.551625 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:35.565447 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:35.582161 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:22:35.596756 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:35.724874 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:35.734760 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:35.734972 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:35.741426 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:35.743284 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:22:35.791283 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:35.814976 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:22:35.830180 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:35.845032 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:35.870021 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:35.885372 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:35.945777 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:35.961187 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:22:35.986125 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:36.024420 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:36.118074 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:36.127641 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:22:36.128522 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:22:36.133688 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:36.239220 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:36.255875 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 08:22:36.322095 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:22:36.329548 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:36.344817 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:36.346766 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:36.359642 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:36.441736 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:36.468267 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:36.472115 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:36.538576 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:36.651645 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:36.660540 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:36.684445 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:36.832819 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 08:22:36.833823 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:36.864032 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:36.884045 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:36.893774 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:36.937554 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:22:36.946758 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:36.951963 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:36.974265 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:36.976157 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:22:36.981554 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:36.982709 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:36.997001 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:22:37.045051 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:37.045203 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:22:37.045845 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:37.049471 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.099085 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:37.143518 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.152216 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:37.164988 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:37.314213 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.350497 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:37.362877 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.376663 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.394330 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:37.418318 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.428351 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:37.442639 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:22:37.449457 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:37.460021 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:37.465723 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:37.473305 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:37.487818 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:37.490552 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:37.540414 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:37.553100 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:37.559484 12925 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.4:32979, user_credentials={real_user=slave}} blocked reactor thread for 45907.5us
W20240502 08:22:37.618522 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:22:37.619976 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:37.623390 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.640130 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.656926 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.680213 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.770666 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.788688 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:37.862644 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.868001 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.932178 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:37.941023 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:37.944243 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:22:37.957049 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:37.957504 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:37.965754 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:37.973752 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:37.997324 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:38.003965 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:38.019780 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:38.032321 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:22:38.044715 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:38.045121 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:38.063838 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:38.107188 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:38.131395 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:38.169363 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:38.184548 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:38.194505 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:38.238438 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:38.240175 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:38.334177 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:38.368723 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:38.408221 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:38.448613 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:22:38.604236 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:22:38.692785 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:38.694738 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:38.694813 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:38.780685 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:38.793648 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:38.830246 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:38.839308 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:38.841636 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:38.864249 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:38.884402 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:38.969489 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:39.148440 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:22:39.183944 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:39.189702 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:39.229694 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:39.230819 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:39.272207 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:39.274936 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:39.275653 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:39.306422 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:39.308604 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:39.313902 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:39.352066 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:39.376420 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:39.413412 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:39.422261 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:39.482713 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:39.532392 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:39.543126 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:39.544580 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:39.576352 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:39.636301 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:39.702064 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:22:39.719297 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:39.738998 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:39.757978 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:39.759188 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:39.764701 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:39.784988 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:39.810572 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:22:39.839345 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:39.855806 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:39.856385 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:39.895992 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:22:39.899364 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:39.902828 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:39.934940 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:39.950428 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:39.975118 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:40.023321 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.029266 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:40.068449 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:40.090656 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:40.174667 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240502 08:22:40.199736 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:40.213155 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:40.217399 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:40.218526 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:40.244124 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:22:40.307986 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:40.327543 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:40.329128 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:40.354542 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 08:22:40.376104 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:40.417866 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:40.429153 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:40.430303 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:40.459710 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:22:40.461911 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:40.489140 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.501284 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.513223 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.545531 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.548825 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.559844 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.561141 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.582880 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:40.603807 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.608086 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.688942 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.704296 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.713132 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:22:40.722688 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:22:40.744248 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:40.830289 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:22:40.835294 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:40.873965 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:22:40.925299 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:40.957242 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:40.957492 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:40.970490 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:40.977775 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:40.989339 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:41.024032 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:22:41.043936 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:41.132797 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:41.160497 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240502 08:22:41.350772 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 08:22:41.351682 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:41.355641 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:41.377208 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:41.511319 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:22:41.587093 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:41.636196 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:41.637264 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:22:41.644467 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:41.646407 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:41.672631 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:22:41.673632 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:41.700788 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:41.704741 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:41.733086 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:41.761551 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:41.785218 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:41.810945 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:41.841944 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 08:22:41.852399 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:41.863143 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:41.961644 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:41.970791 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:41.975678 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:41.989598 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:22:42.103749 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:42.144824 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:42.150892 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:42.159873 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:42.172569 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:42.186693 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:42.194093 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:22:42.206408 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:42.207032 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:42.245386 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:42.280354 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:42.300508 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:42.357616 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:42.394627 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:42.419008 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:42.421404 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:42.427449 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:42.438092 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:42.496395 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:42.565852 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:42.566226 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:42.585355 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:42.587071 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:22:42.614948 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:42.634677 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:42.668218 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:42.670066 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:42.691592 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:42.699719 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:42.718178 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:22:42.720943 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:42.754387 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:42.826992 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:42.846486 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:42.852744 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:42.878407 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:42.891927 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:42.915458 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:42.933231 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:42.951319 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:42.965551 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:42.978214 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:22:43.065759 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:43.072016 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:43.072685 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:43.086333 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:43.090227 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:43.106659 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:43.114531 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:43.133014 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:22:43.168922 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:43.176964 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:43.178256 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:22:43.186681 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:43.208243 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:22:43.272480 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:43.286495 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:43.324858 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:43.339466 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:43.486809 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:22:43.572062 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:43.601014 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:43.603736 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:43.608408 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:43.610108 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:43.615756 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:43.642453 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:43.647362 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:22:43.649839 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:43.654738 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:43.733072 13056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.5:44131, user_credentials={real_user=slave}} blocked reactor thread for 52613.7us
W20240502 08:22:43.869614 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:43.889256 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:43.920850 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:43.959383 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:44.012784 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:44.023702 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:44.124329 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:44.153522 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:44.292713 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:22:44.300051 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:44.314610 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:44.335743 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:22:44.342694 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:44.361783 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:44.381819 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:44.436975 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:44.458097 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:44.518651 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:44.527874 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:44.531612 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:44.536351 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:44.560245 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:44.594950 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:44.640823 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:44.706251 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:44.709650 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:44.798576 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:44.844167 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:22:44.856236 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:44.857910 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:22:44.874269 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:44.881695 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:44.901207 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:22:44.917601 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:44.920079 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:44.933840 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:44.949342 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:44.950706 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:44.972391 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.043169 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.078274 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:45.108781 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.111954 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:45.123732 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:45.140524 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:45.147467 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.159113 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:45.163340 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:45.165894 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.191962 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 08:22:45.246083 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.250295 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.278264 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:22:45.329918 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:45.338850 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:45.340634 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:22:45.361971 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:45.367960 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:22:45.371783 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:45.394632 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.412022 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:22:45.416453 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:45.417263 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:45.434537 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:45.515527 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:22:45.517055 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:45.583910 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:45.585191 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.606910 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:45.646910 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.679656 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.779870 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.805392 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:22:45.850653 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:45.864979 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:22:45.865942 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:22:45.876886 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:45.894482 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:45.912554 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:45.936012 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:22:45.938145 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:45.971771 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:22:45.993381 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:46.014161 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:46.029644 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:46.100730 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:46.101150 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:22:46.162766 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:46.170907 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:46.171780 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 08:22:46.173291 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:46.332944 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:46.333823 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:22:46.391150 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:46.409164 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:46.410877 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:46.417517 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:46.440104 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:22:46.455396 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:46.461247 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:46.466423 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:46.494107 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:46.521224 15688 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 52830.1us
W20240502 08:22:46.522997 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 08:22:46.669307 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:46.727330 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:46.769518 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:46.836336 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:46.889914 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 08:22:46.891170 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:46.922099 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:47.033385 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:22:47.064704 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:47.095654 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:22:47.128033 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:47.134315 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:47.135748 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:22:47.140662 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:47.161064 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:22:47.176796 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:47.198004 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:47.210642 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:47.214195 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:47.240999 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:22:47.252266 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:47.253690 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:22:47.260318 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:47.319005 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:47.373955 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:47.412104 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:47.413527 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:47.467232 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:47.467459 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:47.505147 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:47.509909 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:47.541561 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:47.577634 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:47.589197 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:22:47.670253 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:47.677173 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:47.680294 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:47.683274 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:47.699069 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:47.709594 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:47.756711 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:47.769381 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:47.774163 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:47.774289 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:47.801138 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:47.829381 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:47.857141 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:47.921725 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:47.940443 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:47.941588 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:47.995762 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:48.012640 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:48.017973 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:48.053961 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:48.112622 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:48.125371 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:22:48.161135 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:48.179360 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:48.205884 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:22:48.222527 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:48.228205 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:48.236580 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:48.236975 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:48.246279 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:48.269404 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:22:48.280315 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:48.299747 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:48.413668 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:48.453361 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:48.485065 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:22:48.530189 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:48.541692 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:48.592267 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:48.634987 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:48.696204 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:48.703796 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:22:48.722402 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:48.733438 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:48.823632 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:48.824579 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:48.835805 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:48.846920 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:48.877568 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:48.950152 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:48.956908 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:49.042529 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:49.050205 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:49.117697 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:49.258407 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:22:49.282830 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:49.290385 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:49.306602 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:49.317481 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:49.351428 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:49.355872 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:49.356210 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:49.364913 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:49.384678 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:49.432544 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:49.489867 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:49.494567 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:49.511054 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:49.521962 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:49.577648 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:49.602202 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:49.610034 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:49.776540 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:49.797194 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:49.800050 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:49.866261 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:49.912662 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:49.919469 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:49.931675 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:49.938354 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:49.998632 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:50.004215 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:50.021682 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:50.038189 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:50.048535 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:22:50.057713 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:50.079187 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:50.094606 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:50.105587 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:50.107401 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:50.133330 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:22:50.136919 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:50.151674 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:50.156903 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:50.178694 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:50.196007 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240502 08:22:50.287046 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:50.341370 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:50.368431 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:50.381927 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:50.404558 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:50.424854 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:50.430842 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:50.511724 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:50.522157 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:50.544654 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:50.548738 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:50.548835 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:22:50.589401 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:50.594167 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:50.613018 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:50.644804 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:50.654778 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 08:22:50.721200 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:22:50.724710 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:50.726327 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:50.738906 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:22:50.745065 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:50.796969 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:50.812548 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:50.824368 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:50.856063 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:22:50.932332 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:50.955783 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:22:50.975587 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:22:50.976536 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:51.097990 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:51.114491 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:51.140928 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:51.149092 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:51.165184 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:51.193073 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:51.203554 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:51.206943 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:51.229111 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:51.235561 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:51.263298 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:51.309123 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:51.332753 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:22:51.336900 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:51.456642 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:51.481671 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:51.600888 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 08:22:51.605242 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:51.605865 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:51.605983 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:51.687422 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:51.735810 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:51.740991 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:51.744544 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:51.750308 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:51.768105 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:51.799268 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:51.814709 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:51.844200 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:51.914949 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:51.915963 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:51.982245 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:52.020924 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:52.065832 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 08:22:52.099086 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:52.100128 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:52.124868 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:52.150719 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:52.151935 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:52.157121 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:52.172240 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:52.180127 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:52.189689 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:52.194710 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:52.221601 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:52.221948 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:52.245529 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:22:52.255995 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:52.262748 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:52.271032 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:52.339181 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:52.425236 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:52.508358 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:52.513046 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:52.562996 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:52.579484 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:52.601063 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:52.602030 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:52.650534 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:52.682227 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:52.689038 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:52.768585 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:52.838387 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:52.847199 15418 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.7:38851, user_credentials={real_user=slave}} blocked reactor thread for 57504.4us
W20240502 08:22:52.850771 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:52.899668 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:52.914216 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:52.917218 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:52.929931 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:52.937207 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:52.958689 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:52.979651 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:53.001757 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:53.002916 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:53.018704 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:22:53.036276 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:53.037892 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:22:53.038403 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:53.047029 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:53.167979 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:53.180460 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:53.202634 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:53.218999 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:22:53.251740 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:53.253926 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:53.273655 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:53.319468 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:53.344938 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:22:53.368060 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:53.378365 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:53.381707 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:53.403501 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:53.409447 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:53.412734 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:53.428452 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:53.483795 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:53.571825 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:53.576963 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:53.600469 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:53.645181 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:53.763124 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:53.876405 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:53.909754 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:53.922322 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:53.939713 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:53.950073 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:53.968786 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:22:53.984172 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:53.999194 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:54.008591 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:54.043282 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:54.154778 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:54.195096 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:54.196568 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:54.277055 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:54.342046 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:54.397498 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:54.439108 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:54.467458 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:54.479436 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:54.496224 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:54.496575 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:54.543790 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:54.553580 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:54.600070 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:54.604948 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:54.623476 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:54.683421 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:54.691515 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:54.715987 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:54.773886 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:54.792740 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:54.794071 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:54.806614 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:22:54.822872 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:54.842845 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:54.873370 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:54.901403 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:54.959553 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:22:55.005156 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:55.025712 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:55.033739 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:55.036106 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:55.045228 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:55.076788 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:55.083767 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:55.113512 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:55.119736 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:55.134940 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:22:55.140630 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:55.161916 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:55.177989 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:55.199271 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:55.210176 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240502 08:22:55.222537 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:55.222769 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:55.250978 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:55.302690 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:55.324173 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:55.330888 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:55.375782 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:55.397588 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:55.499312 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:55.502446 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:55.521155 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:55.601763 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:55.658068 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:22:55.674727 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:55.710563 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:55.750101 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:55.758973 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:55.759871 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:22:55.769850 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:55.787981 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:55.808670 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:55.810735 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:55.830981 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:55.839401 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:55.847189 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:55.858718 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:55.891397 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:55.914911 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 08:22:55.949834 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:55.955624 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:55.972782 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:56.046800 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:22:56.051052 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:56.092212 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:56.183913 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:22:56.231580 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:56.237041 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:56.242260 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:56.244377 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:56.263885 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:56.264293 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:56.271451 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:56.327690 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:56.343166 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:22:56.359875 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:56.378540 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:56.418645 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:56.534667 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:56.585002 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:56.668529 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 08:22:56.718602 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:56.749375 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:56.763801 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:56.793004 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:56.797377 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:56.858134 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:56.864190 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:56.866413 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:22:56.899132 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:56.906015 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:56.921761 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:56.941162 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:56.972805 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:57.079785 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:57.083132 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 08:22:57.114627 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:57.125931 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:57.143985 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:57.151690 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:57.223399 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:57.240100 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:57.325198 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:57.338883 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:57.355314 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:57.358755 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:57.371747 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:57.394645 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:57.396826 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:22:57.402664 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:57.420358 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:57.422096 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:57.428926 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:22:57.434110 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:57.501058 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:57.511206 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:57.511495 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:57.511974 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:57.512080 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:57.569821 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:57.574692 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:57.693642 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:57.706612 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:57.712208 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:57.722792 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:57.761034 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:57.822333 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:57.845098 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:57.862479 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:57.873507 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:57.908629 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:57.934960 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:57.952832 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:57.953083 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:57.958191 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:58.015383 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:58.031474 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:58.032008 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:58.076856 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:58.085276 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:58.114049 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:22:58.182888 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:58.219769 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:58.231117 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:58.251027 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:22:58.274000 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:58.300032 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:58.307436 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:58.323937 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:22:58.380255 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:58.402557 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:58.525823 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:58.658877 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:22:58.696460 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:58.718318 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:58.719710 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:22:58.729905 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:22:58.740818 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:58.775444 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:22:58.794358 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:58.796064 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:58.827641 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:58.828089 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:22:58.857651 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:58.883246 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:59.052862 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:22:59.063656 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:59.067287 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:22:59.178591 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:59.184465 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:22:59.221798 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:59.227116 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:59.231470 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:59.236546 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:59.261950 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:59.281394 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:22:59.283160 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:22:59.303586 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:59.374820 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:59.405695 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:59.500775 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:22:59.681500 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:22:59.696511 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:22:59.702898 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:59.712410 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:22:59.713367 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:22:59.718943 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:22:59.750875 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:22:59.760962 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:22:59.766016 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:59.775080 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:22:59.782737 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:22:59.803671 15419 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.3:46757, user_credentials={real_user=slave}} blocked reactor thread for 46785.5us
W20240502 08:22:59.809808 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:22:59.818046 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:22:59.857383 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:22:59.870553 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:59.896229 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:22:59.911249 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:22:59.911867 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:22:59.933754 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:22:59.965726 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:22:59.970558 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:23:00.022727 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:00.025617 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:00.090389 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:00.092883 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:23:00.094313 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:23:00.134377 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:23:00.175926 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:23:00.187325 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:23:00.214111 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240502 08:23:00.262497 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:23:00.288946 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:00.290583 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:00.293375 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:23:00.315618 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:00.323127 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:23:00.328195 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:23:00.330857 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:00.341856 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:00.372967 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:00.375556 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:00.380368 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:00.430893 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:00.478554 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:00.534581 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:00.536278 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:00.587491 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:23:00.674921 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 08:23:00.758275 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:00.761477 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:23:00.770596 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:00.842253 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:00.867220 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:23:00.869673 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:00.872975 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:00.876174 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:23:00.899246 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:00.916244 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:00.921996 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:00.953564 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:23:00.954484 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:23:00.970350 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:00.982295 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:23:00.986081 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:01.046177 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:01.059837 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:23:01.084574 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:01.166044 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:23:01.167295 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:23:01.195120 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:01.209923 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:01.346220 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:01.369871 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:01.394537 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:23:01.557689 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:01.590062 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:01.601570 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:23:01.610616 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:01.614856 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:01.644654 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:01.660514 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:01.684723 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:01.720185 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 08:23:01.728093 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:01.740149 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:23:01.753541 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:01.768718 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:01.769410 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:01.838804 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:01.843953 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:01.869951 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:23:02.071076 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:02.087325 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:02.097728 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:02.100802 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:23:02.106053 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:02.121557 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:02.133227 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:02.150092 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:23:02.162961 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:02.163213 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:02.171988 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:02.193120 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:02.238263 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:02.241058 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:02.272197 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 08:23:02.348392 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:02.422385 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:02.426287 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:02.483120 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:02.498365 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:23:02.509281 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:02.519099 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:02.548805 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:02.557593 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:02.566565 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:02.574271 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:23:02.640472 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:02.649773 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:23:02.658977 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:02.671437 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:23:02.692567 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:02.716779 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:02.741403 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:02.746449 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:02.777449 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:23:02.779441 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:02.793896 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:02.798118 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:02.886031 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:02.895902 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:02.925225 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:02.925607 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:02.928783 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:02.986079 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:03.002755 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:03.025715 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:03.092592 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:23:03.171890 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:03.174858 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:03.192965 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:03.198215 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:03.218504 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:03.219188 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:03.220309 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:03.241291 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:03.241452 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:03.244693 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:03.329447 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:03.417980 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:03.484716 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:03.488847 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:03.560109 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:03.573379 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:03.592948 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:03.618178 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:03.671128 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:23:03.674957 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:03.676115 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:23:03.708518 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:03.709112 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:03.739180 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:03.766224 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:03.806851 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:03.813613 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:23:03.844324 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:03.857322 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:03.967474 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:03.976290 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:04.031790 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:04.051522 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:04.061466 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:04.190331 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:04.252321 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:04.303644 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:04.357266 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:04.361145 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:23:04.452418 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:04.490100 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:04.514307 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:04.555474 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:04.556720 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:04.565147 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:04.594480 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:23:04.606405 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:04.638599 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:04.668695 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:23:04.690887 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:04.701366 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:04.725800 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:04.744925 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:04.766875 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:04.834767 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:04.847783 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:04.875792 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:04.878764 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:04.892596 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:04.904464 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:04.962648 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:04.981637 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:04.991766 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:23:05.026538 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:05.029701 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:05.038064 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:05.068410 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:05.108867 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:05.133735 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:23:05.135193 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:05.142939 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:05.159309 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:05.163273 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:05.199282 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:05.215624 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:05.244503 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240502 08:23:05.304872 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:05.332113 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:05.380254 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:05.383275 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:05.400825 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:05.424829 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:05.425781 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:05.478255 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:05.564455 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:05.571045 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:05.624998 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:05.643915 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:05.651872 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:05.659211 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:05.671368 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:05.674042 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:05.680996 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:23:05.697432 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:05.716872 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:23:05.767949 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:05.870270 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:05.877386 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:05.884624 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:05.977108 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:05.997184 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:06.008601 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:06.041862 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:23:06.064236 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:06.077205 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:06.084154 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:06.090354 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:23:06.095114 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:06.111104 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:06.122898 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240502 08:23:06.132043 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:06.157559 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:06.170449 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:06.183156 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:06.207991 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:06.277154 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:06.310761 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:06.348975 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:06.497123 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:06.504869 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:23:06.518018 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:06.522936 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:06.527706 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:06.606285 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:23:06.620991 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:06.632722 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:06.672622 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:23:06.686805 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:06.715615 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:06.721810 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:23:06.725344 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:06.731740 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:23:06.732502 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:06.765436 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:06.767408 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:06.799649 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:06.888053 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 08:23:06.936698 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:06.967978 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:07.011010 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:07.147629 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:07.180227 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:07.183672 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:07.189659 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:07.196061 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:07.361672 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 08:23:07.363730 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:23:07.393251 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:07.419673 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:07.429162 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:07.432240 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:07.439611 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:07.448071 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:07.451226 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:23:07.488863 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:07.520884 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:07.560021 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:07.564332 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:07.586038 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:07.595805 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:07.597064 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:07.609808 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:07.635326 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:07.638172 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:07.643868 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:23:07.660420 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:07.717630 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:07.849813 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:07.921859 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:07.933941 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:23:07.937323 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:07.937346 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:07.940852 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:07.977057 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:07.983525 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:23:07.998848 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:08.007225 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:08.018289 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:08.027769 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:23:08.032390 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:08.070164 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:08.116600 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:08.123054 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:08.132145 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:08.235365 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:08.240139 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:08.339036 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:08.342983 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:08.348587 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:08.469756 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:08.472960 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:08.477034 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:08.529938 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:08.532752 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:08.575560 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:23:08.584167 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:08.591146 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:08.620640 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:08.650080 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:08.666225 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:23:08.670753 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:08.693331 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:23:08.702440 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:08.722867 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:08.845719 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:08.882218 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:08.886233 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:09.008455 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:09.020107 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:09.030309 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:09.040549 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:23:09.051353 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:09.060822 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:09.062673 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:09.068130 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:09.086407 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:09.115809 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:09.128511 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:09.128994 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:23:09.133626 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:09.253948 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:09.278548 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:09.355458 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:09.362061 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:09.487769 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:09.496193 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:09.502164 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:09.564522 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:09.571444 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:09.604609 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:09.622610 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:09.637102 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:09.637892 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:09.640437 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:09.643568 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:09.670209 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:09.681246 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:09.706528 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:09.882485 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:09.907699 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:09.955349 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:09.956053 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:10.020925 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:10.040406 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:10.046447 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:10.110935 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:10.111243 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:10.139778 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:10.150802 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:10.216662 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:10.253003 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:23:10.273938 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 128 similar messages]
I20240502 08:23:10.306710 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:10.311581 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:10.329537 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:10.342259 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:10.349418 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:10.350497 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:10.375167 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:10.401541 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:10.403431 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:10.428609 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:10.454713 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:10.461545 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:10.517521 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:10.568064 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:10.581956 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:10.598948 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:10.608098 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:10.624189 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:10.652972 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:10.679838 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:10.740303 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:10.799090 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:10.805017 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240502 08:23:10.814908 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:10.850410 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:10.872535 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:10.890815 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:10.919118 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:10.956475 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:10.962757 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:10.978327 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:10.990015 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:11.058897 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:11.064232 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:11.113373 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:23:11.190639 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:11.222549 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:11.238631 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:11.239374 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:11.271945 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:11.282634 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 08:23:11.284803 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:11.294345 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:23:11.324471 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:11.360875 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:11.370071 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:11.373099 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:11.373919 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:23:11.377068 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:11.391237 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:11.421607 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:11.452493 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:11.458619 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:11.458643 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:11.473037 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:11.486923 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:11.509896 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:23:11.529755 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:11.707952 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:11.744848 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:11.760790 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:11.778702 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:11.838320 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:23:11.893687 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:11.909173 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:11.951710 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:11.952625 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:11.965813 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 08:23:11.965839 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:11.973510 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:11.974526 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:11.989959 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:23:12.041350 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:12.054004 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:12.058068 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:12.087541 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:12.210358 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:12.301894 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:12.326391 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:12.358966 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:12.381045 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:12.460675 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 08:23:12.463002 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:12.487253 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:23:12.489076 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:12.489435 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:12.531625 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:12.554167 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:12.560812 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:12.582254 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:12.582769 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:12.609020 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:12.642642 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:12.653101 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:12.754079 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:12.821249 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:12.830165 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:12.853505 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:12.900612 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:12.911180 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:12.955505 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:23:12.972854 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:12.976564 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:12.977788 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:12.988394 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:13.048005 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:13.067520 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:13.114055 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:13.134264 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:13.189266 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:13.205181 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:13.225544 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:13.243193 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:13.245968 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:13.263685 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:13.268445 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:13.311120 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:13.323792 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:13.330758 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:13.398516 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:13.415575 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:13.472101 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:13.489715 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:13.509935 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:13.529575 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:13.629987 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:13.744918 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:23:13.756096 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:13.769816 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:13.770053 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:13.783794 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:13.785235 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:13.799141 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:13.809348 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:13.819569 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:13.827105 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:13.872128 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:13.892118 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:13.911756 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:13.921406 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:13.930135 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:13.963739 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:14.019544 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:14.118546 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:14.132688 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:14.141276 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:14.164616 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:23:14.271948 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:14.272728 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:14.273317 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:14.278595 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:14.297282 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:14.287220 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:14.313386 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:23:14.341480 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:14.347600 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:14.366274 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:14.406761 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:14.470805 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:14.564376 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:14.599763 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:14.608902 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:14.627573 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:23:14.718122 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:14.751561 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:14.757401 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:23:14.778723 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:14.785409 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:14.839218 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:14.857753 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:14.868558 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:23:14.870455 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:14.889376 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:23:14.915323 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:14.940094 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:14.965243 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:14.976717 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:15.080199 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:15.168787 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:15.179278 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:15.244160 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:15.261036 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:23:15.266723 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:15.278052 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 129 similar messages]
I20240502 08:23:15.287932 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:15.297087 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:15.305712 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:15.346312 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:23:15.357038 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:15.401364 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:15.423414 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:15.463321 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:23:15.478598 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:15.513384 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:15.521895 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:23:15.523015 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:15.587997 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:15.588223 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:15.597186 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:15.603101 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:15.648597 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:15.703928 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:15.718180 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:15.861601 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:15.873575 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:15.873701 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:15.893290 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:15.939590 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:15.943261 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:16.015162 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:16.025956 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240502 08:23:16.053354 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:16.061841 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:23:16.079962 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:16.084189 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:16.084353 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:16.084430 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:16.110250 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:16.129025 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:23:16.129851 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:23:16.149075 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:16.163264 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:23:16.191237 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:16.219797 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:16.273227 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:16.332661 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:16.370640 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 08:23:16.379956 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:16.401832 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:16.468237 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:16.492486 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:16.521292 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:16.604569 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:23:16.614629 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:16.635860 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:16.655922 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:16.663362 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:16.717583 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:16.719285 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:16.731889 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:23:16.733768 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:16.780274 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:16.780823 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:16.800844 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:23:16.816543 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:16.817922 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:16.841043 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:16.926908 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:16.989317 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 08:23:16.990154 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:17.033624 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:17.054157 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:17.169775 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:17.230070 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:17.240542 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:23:17.241860 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:17.248392 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:17.256772 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:17.262249 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:23:17.294312 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:17.325124 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:17.326911 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:23:17.330201 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:17.360755 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:17.406591 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:17.455107 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:17.458949 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:17.510901 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 08:23:17.610234 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:17.612711 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:17.614616 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:17.713696 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:17.725338 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:17.761468 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:17.782387 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:17.794742 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:17.849208 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:17.880419 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:23:17.884596 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:17.910094 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:17.912649 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:17.929759 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:17.954409 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:23:17.960616 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:17.962038 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:17.967573 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:17.996397 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:23:17.997558 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:17.997704 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:18.073966 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:18.130828 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:18.133908 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:18.139055 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:18.153649 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:18.203141 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:18.224026 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:18.224205 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:23:18.373204 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:18.392802 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:18.397727 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:18.402552 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:18.414235 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:23:18.435214 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:18.495792 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:18.497332 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:18.522598 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:18.532303 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:18.539649 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:18.578512 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:18.606417 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:18.649778 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:18.667658 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:18.724776 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:18.759505 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:18.922583 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:18.963678 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:18.972404 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:18.980293 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:18.988289 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:19.035992 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:19.037948 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:23:19.068945 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:19.092772 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:23:19.123713 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:19.140164 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:19.159802 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:19.182653 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:19.211930 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:19.226948 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:19.242156 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:19.311177 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:19.312821 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:23:19.320863 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:19.328456 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:19.372699 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:19.416702 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:19.428593 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 08:23:19.445156 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:19.544456 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:19.671949 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:19.724251 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:19.731264 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:19.769296 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:19.791697 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:19.795325 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:19.805533 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:19.850593 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:19.857610 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:23:19.881438 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:19.898496 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:19.898900 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:19.900090 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:19.904163 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:19.933717 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:20.052101 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:20.061724 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:20.112813 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:20.166028 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:20.191258 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:20.240942 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:20.283118 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:20.295668 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:20.298630 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:20.341559 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:20.342499 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:23:20.343983 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:20.359861 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:20.369120 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:20.382930 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240502 08:23:20.388545 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:20.390542 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:20.423931 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:20.434031 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:20.509805 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:20.547098 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:20.590713 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:20.645283 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:20.660002 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:20.694797 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:20.699769 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:20.700110 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:20.796725 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:20.803027 12927 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.8:46303, user_credentials={real_user=slave}} blocked reactor thread for 81752.5us
W20240502 08:23:20.803905 12926 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.5:44131, user_credentials={real_user=slave}} blocked reactor thread for 78032.8us
W20240502 08:23:20.805244 12925 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.5.7:38851, user_credentials={real_user=slave}} blocked reactor thread for 80767.9us
I20240502 08:23:20.857838 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:20.872814 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:20.893587 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:20.910306 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:20.912837 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:20.932340 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:20.972977 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:21.000329 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:21.004289 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:21.009058 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:21.038065 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240502 08:23:21.039388 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:21.073438 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:21.083756 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:21.102684 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:21.153026 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 08:23:21.191001 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:21.263204 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:21.287324 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:21.351855 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:21.361537 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:21.399235 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:21.421933 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:21.447268 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:21.463635 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:21.475530 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:21.477015 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:21.484242 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:21.488389 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:23:21.489122 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:21.502149 15421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:23:21.503598 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:21.533630 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:21.541352 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:21.541464 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:21.550415 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:21.563652 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:21.659147 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:21.719310 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:21.738692 12666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:23:21.767364 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:21.789211 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:21.899746 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:21.950378 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:21.979004 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:22.011204 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:22.040038 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:22.047705 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:22.087769 15690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 08:23:22.113632 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:22.130860 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:22.132810 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:22.133987 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:22.143337 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:22.173934 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:22.204699 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:22.205605 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:22.227723 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:22.259040 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:22.259905 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:22.417287 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:22.468765 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:22.504982 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:22.508387 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:22.525892 13058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 08:23:22.542433 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:22.581218 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:22.587671 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:22.684803 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:22.697363 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:22.705821 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:22.722661 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:22.725972 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:22.728142 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:22.734592 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:22.740152 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:22.773639 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:22.797276 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:22.828198 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:22.881644 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:22.909873 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:23:22.959674 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:23.030499 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:23.041159 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:23.085875 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:23.087611 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:23.131772 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:23.151845 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:23.160511 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:23.192274 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:23.205322 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:23.217548 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:23.224172 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:23.225582 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:23.235960 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:23.240123 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:23.274274 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:23.302918 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:23.308415 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:23.313177 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:23.333020 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:23.365742 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:23.412196 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:23.478928 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:23.549357 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:23.573776 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:23.619439 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:23.634404 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:23.654863 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:23.658803 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:23.658975 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:23.661877 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:23:23.678759 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:23.693397 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:23.743147 15822 consensus_peers.cc:485] T 5fcbd19e0a9b49eaa3e403683fe6b34b P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:23.755486 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:23.808133 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:23.901564 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:23.919615 12798 consensus_peers.cc:485] T 039268a812b64d88a922d0399074647c P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:23.921723 15690 consensus_peers.cc:485] T 6a9d9100dbaa4861ba1d2a1455f9756d P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:23.975208 12666 consensus_peers.cc:485] T 4e6a660632634e3c9abde21dbf84dfc0 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:23.982034 15822 consensus_peers.cc:485] T a9f90ed1e4e04c93995a5abba79ecb34 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:23.988898 15822 consensus_peers.cc:485] T 9e51af7112424e52a9e9c3200a30f86d P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:24.109257 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:24.118907 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:24.137841 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:24.150298 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:24.177232 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:24.195183 12928 consensus_peers.cc:485] T c63cca9cd28f4d30bf901e1de1cd6461 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:24.201915 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:24.234728 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:24.243710 15822 consensus_peers.cc:485] T 3c58721561064e97909a0caddc47a1f0 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:24.280189 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:24.293074 15421 consensus_peers.cc:485] T c9ad7bf820c84aae9a8875fb92f8e5d5 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:24.319507 15690 consensus_peers.cc:485] T ea85ef42ac714074a10f2a5ece4ddf8e P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:24.357260 15822 consensus_peers.cc:485] T ad1bea423b354df38e9e0100c1560e08 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:24.371706 12798 consensus_peers.cc:485] T 8de012363e4f4455acd96cb674ab0c4b P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:24.375341 15421 consensus_peers.cc:485] T 42e0e206eca841fd8dfa7146f8f2d890 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:24.414671 12928 consensus_peers.cc:485] T 553cec4eb28341dba84afc79b540ae4b P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:24.434583 15690 consensus_peers.cc:485] T 156838b04be24853897a4fabb5916029 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:24.699000 13058 consensus_peers.cc:485] T ad53a92fbcc041378bb1958c65e997db P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:24.763345 15421 consensus_peers.cc:485] T 9a55a182e1f54db698ba8060eea80da0 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:24.779006 15822 consensus_peers.cc:485] T 0e882f58c7f344f7929b8a30bd65fa44 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:24.812350 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240502 08:23:24.817974 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:24.833204 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:24.833914 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:24.847640 12798 consensus_peers.cc:485] T bb73ec8c37464c8ba97ac2372667525a P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:24.860565 15421 consensus_peers.cc:485] T 30001d4bd6e943f88d28894ef59cbe18 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:24.899854 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 08:23:24.901343 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240502 08:23:24.966178 12798 consensus_peers.cc:485] T d0dede000b7943439f7b8455ea370e14 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:24.969882 15822 consensus_peers.cc:485] T 939120da41cd41cb83ef225b9942f82e P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:24.982445 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:24.991303 12666 consensus_peers.cc:485] T 5b29a9d8d26b4a388c89fb8e3a0fefe3 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:23:24.995951 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:25.004720 12798 consensus_peers.cc:485] T e9081ea0b98443d38f42c14cd626f0d0 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:25.011214 15421 consensus_peers.cc:485] T 02ec4a224738406185f93fccb3b9320e P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:25.026242 15690 consensus_peers.cc:485] T aadd5844b106485e95ecf6fa923cd6ea P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:25.032088 13058 consensus_peers.cc:485] T 432a6e076bf4401bb81903d91de38a57 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:25.033910 12666 consensus_peers.cc:485] T 3c37ff40706043e9adc4da4a6da1947a P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:25.130194 15822 consensus_peers.cc:485] T cb05ad19b6a84f35a6853db5e3cabf9a P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:25.138072 12928 consensus_peers.cc:485] T 7082603ad2d6450e8863cd53b1897b2e P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:25.193286 12666 consensus_peers.cc:485] T 809af1a2878d4cba9ab2f18c11a621a8 P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:23:25.329522 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:25.351521 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:25.355019 15421 consensus_peers.cc:485] T c8b77a0b50464cff9c1879d57fc06ed9 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:23:25.375447 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:25.378789 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:25.387670 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
I20240502 08:23:25.394196 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:25.409389 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 132 similar messages]
I20240502 08:23:25.412039 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240502 08:23:25.432420 15690 consensus_peers.cc:485] T 65cf482906a646aabd59f22900ece16f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:25.449260 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:25.550168 13058 consensus_peers.cc:485] T a5af24c28aa04f72844744771f6b7249 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:23:25.565366 15822 consensus_peers.cc:485] T 01b4100f5ab3405cabd93cff75f61822 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:25.597347 15421 consensus_peers.cc:485] T 7d4a5f0c5c184b7f975f7bcc59cb6b4c P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:25.643412 12798 consensus_peers.cc:485] T 8774c037b901461684d96460e663d17e P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:25.654832 15822 consensus_peers.cc:485] T 1b7190137f2949de8df2cdf447802db1 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:25.669075 15822 consensus_peers.cc:485] T 24c29ddfac024726ad9ee74d77c115f9 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:25.687065 13058 consensus_peers.cc:485] T fd0a5745bdba436e848ba33029ef01b4 P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:25.728528 12928 consensus_peers.cc:485] T 83e0c78cb2374ba483315ca8a827e532 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:25.738593 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:25.750300 12928 consensus_peers.cc:485] T 8f2cda653a044a41bbf61dff7bb29fe2 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:25.756130 15690 consensus_peers.cc:485] T 18c58e2a9a814bcb8ff6bb9e3011084f P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:23:25.804033 15865 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 08:23:25.810650 12708 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240502 08:23:25.826925 15733 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240502 08:23:25.842355 15690 consensus_peers.cc:485] T 815d79a2bbba49ea9e135d94c57a3390 P 9333cb10227b4801b4fa7c24e22e131b -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:23:25.850749 12970 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:25.853785 15468 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 08:23:25.854487 12842 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 08:23:25.857223 13102 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240502 08:23:25.965869 12666 consensus_peers.cc:485] T 6a72d875fcb54315af393e31e743d2ae P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:25.969277 15421 consensus_peers.cc:485] T f3f9adebeded4492a2b18b9b60de1c79 P db7893bb29a047b3b82cc60bfd8930a3 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:26.033380 15822 consensus_peers.cc:485] T c60b24cf2d4740ae9b8829e288bc2530 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:26.039373 15822 consensus_peers.cc:485] T 188e9a523fee46a08a4c6180058ed923 P 10f0e36203914b85b77e78609ffced98 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:23:26.052920 12666 consensus_peers.cc:485] T ab808742ce8746ae8e9b8fbff435e7be P 29b035434087435cbaf4ebcac98eedee -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:23:26.074194 12798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 08:23:26.106232 12928 consensus_peers.cc:485] T 84204315ebcf4ccbb0a62de8daa5cb23 P 60b6e265e7754b7187307d05ef405dd4 -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:23:26.142402 13058 consensus_peers.cc:485] T 7a8e5440619a4cb0b40dd286862184cd P 6a0b05f3250d415590b0edf6b44fbe1f -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn't send request to peer 8d45ab45b8eb4d03a4bb9cb88b78a98b. Status: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:23:26.199384 12798 consensus_peers.cc:462] T 8c28e9d3d29548a894b520ae7b324129 P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111)
W20240502 08:23:26.247347 12928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.5.6:44983: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:23:26.291683 12798 consensus_peers.cc:485] T 495ea7de81d34b50a92f6cb52795fadf P e1dd14f7ca764a1490df419f13f2804a -> Peer 8d45ab45b8eb4d03a4bb9cb88b78a98b (127.6.5.6:44983): Couldn'